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Overview 


While over half of all community college students are judged to need developmental (or remedial) 
reading, composition, and/or mathematics classes, these courses — which students are often required 
to complete before they can enroll in courses that confer credit toward a degree — typically present 
major roadblocks to student progress. To address this issue, the Developmental Education Initiative 
(DEI) was created in 2009. Fifteen highly diverse community colleges that had been early participants 
in Achieving the Dream, a national community college refomi network, each received a three-year 
grant of $743,000 to scale up existing interventions or establish new ones that would help students 
progress through developmental courses more quickly and successfully. The colleges typically 
identified two or three “focal strategies” — most often, student support services and new instructional 
strategies — for achieving these goals. This second and final report from the evaluation relies on both 
qualitative and quantitative data to examine the implementation of these focal strategies. 

The report finds that, across the colleges, the percentage of incoming developmental students 
participating in at least one focal strategy more than doubled, rising from 1 8 percent in fall 2009 to 
41 percent in fall 2011. Resource adequacy, communication, engagement, and a departmentwide or 
institutionwide commitment to a particular instructional practice all facilitated scale-up. At the same 
time, colleges generally expected to reach many more students with their reforms than they actually 
did. Factors that worked against greater scale-up sometimes reflected competing values and goals: 
institutional reluctance to impose mandates about how students should leam and instructors teach, 
students’ own wishes and priorities, a perceived need to scale back when strategies appeared to be 
ineffective, and a desire to evaluate the strategies’ apparent effectiveness before moving forward. 

A rigorous impact study was not part of the evaluation. Instead, outcomes for focal strategy partici- 
pants were compared with outcomes for nonparticipants, and outcomes for pre-DEl cohorts of 
students were compared with outcomes for students who enrolled after the DEI began. While the 
results cannot be regarded as conclusive, the two different analytic approaches yield similar findings: 
Most often, there was no statistically significant difference between the two groups being compared. 
When there was such a difference, students were much more likely to benefit from the DEI strate- 
gies than to be harmed by them. 

The DEEs influence on participating colleges extends beyond the focal strategies. The colleges used 
DEI monies to support policy changes and other programmatic reforms as well as to fund both off- 
site conference attendance and on-campus professional development on a broad range of topics 
related to developmental education. The DEI stimulated wider discussions about student success and 
campus priorities, and some DEI innovations will carry over into future initiatives. 
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Preface 


As America climbs out of the Great Recession, the workers most likely to be left behind are 
those without postsecondary credentials. Community colleges play an important role in supply- 
ing students with the education and training that they need to succeed in the labor market. But 
many students enrolling in community college lack the academic skills deemed necessary to do 
college-level work. They are required, instead, to take developmental (remedial) classes before 
embarking on coursework that confers credits and leads to a degree. Unfortunately, such 
students all too often do not complete their prescribed developmental coursework, and they 
“stop out” or drop out of college altogether. 

The Developmental Education Initiative (DEI), funded by the Bill & Melinda Gates 
Foundation and by Lumina Foundation, was an effort to remedy this situation. Fifteen commu- 
nity colleges were selected to expand preexisting interventions or put in place new ones directed 
toward helping students move through developmental coursework more quickly and more 
successfully. MDRC was asked to assess the degree of scale-up that took place, to examine the 
outcomes associated with the interventions, and to identify the conditions that facilitated or 
constrained scale-up efforts. 

This study shows that colleges were, indeed, able to expand the reach of their interven- 
tions — although, in most cases, not to the extent that they had hoped. While the evaluation 
design was not rigorous enough to provide conclusive evidence, there is reason to think that 
some of the interventions improved student outcomes. Finally, the report singles out a number 
of factors that led to successful scale-up. It is hoped that its lessons will assist other colleges as 
they seek to help more underprepared students to attain the skills and credentials that have 
increasingly become key to workplace success. 

Gordon L. Berlin 
President 
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Executive Summary 


There is wide agreement that the well-paying jobs of the future will require postsecondary 
credentials. But for many students attending community college, developmental (or remedial) 

classes in reading, composition, and/or mathematics the courses that students often must 

complete before they can enroll in courses that confer credit toward a degree pose an often- 

insuperable barrier to progress. While over half of all community college students are judged to 
need at least one developmental class, the majority of students who are referred to developmen- 
tal education do not complete their prescribed sequence of remedial courses, much less persist 
and obtain a diploma or certificate. 

To address this issue, the Bill & Melinda Gates Foundation created the Developmental 
Education Initiative (DEI) in 2009; Lumina Foundation for Education funded the evaluation. 
Fifteen colleges that had been early participants in Achieving the Dream (AtD): Community 
Colleges Count, a national community college reform network dedicated to evidence-based 
decision-making, were selected to receive grants of $743,000 each over a three-year period. The 
institutions are highly diverse in size, location, and the characteristics of the students they serve. 
The purpose of the DEI grants was to enable the colleges to scale up existing interventions, or 
establish new ones, that would help students to progress through developmental courses more 
rapidly and more successfully or to bypass these courses altogether. DEI funding also financed 
state policy teams that sought to influence state higher education legislation and policies. MDC, 
a North Carolina-based nonprofit organization, was selected as managing partner of the demon- 
stration and in this role monitored and assisted the colleges, organized communications, and 
convened regular meetings of demonstration participants. 

Six other organizations made up the partnership that provided leadership and support 
for the colleges. Among these, MDRC, a nonprofit, nonpartisan policy research organization, 
was asked to evaluate the demonstration, with the Community College Research Center 
(CCRC) at Teachers College, Columbia University, serving as evaluation partner. The directive 
to the evaluators was to examine the implementation of the DEI at the participating colleges. 
This report — the second and final report from the evaluation 1 — relies on a combination of 
qualitative data (primarily interviews with key personnel conducted during the course of site 
visits to all 15 institutions and through periodic telephone calls with project directors) and 


'For the previous report, see Janet Quint, D. Crystal Byndloss, Herbert Collado, Alissa Gardenhire, Asya 
Magazinnik, Genevieve Orr, Rashida Welbeck, and Shanna S. Jaggars, Scaling Up Is Hard to Do: Progress 
and Challenges During the First Year of the Achieving the Dream Developmental Education Initiative (New 
York: MDRC, 2011). 
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quantitative data (infonnation on participation and on student outcomes that the colleges 
regularly collected). It addresses three main questions: 


• To what extent did the colleges scale up their chosen developmental educa- 
tion reforms to serve more students? 

• What factors affected the colleges’ ability to expand their programs and 
practices? 

• To what extent were the colleges’ strategies associated with improvements in 
student outcomes? 

The report also considers ways that participation in the DEI influenced the colleges more 
broadly. For these reasons, it may be of interest to other colleges looking to scale up reforms 
(especially reforms that are related to instruction and the provision of student supports), as well 
as to funders concerned about how best to support community colleges in bringing promising 
ideas to scale. 


The Initiative’s Premises and On-the-Ground Realities 

The funders’ original goal was for the colleges to expand strategies that the colleges had devel- 
oped as Achieving the Dream sites and that, according to the colleges’ internal evaluations, had 
demonstrated success. Early on, however, the funders agreed that colleges should be allowed to 
develop new initiatives as well as to scale up existing interventions. Soon after the demonstration 
was launched, colleges were advised to identify up to three “focal strategies” that could be 
expected to serve increasing numbers of students over time and on which the evaluation would 
center. As it turned out, the focal strategies that the colleges proposed were almost evenly 
divided between ones that were new and ones that were scaled up from existing interventions. 

Previous efforts to scale up interventions at community colleges had largely gone unex- 
amined, and learning what it takes to expand promising interventions was a key demonstration 
goal. An initial assumption was that funding constraints were a fundamental impediment to 
scaling and that, with these eliminated through generous grants to the DEI institutions, college 
administrators — given modest technical assistance — would have the capacity to lead major 
scale-up efforts. Within the first year, however, it became apparent that a variety of other 
barriers stood in the way of scaling, and MDC was called on to provide more coaching and 
additional information-exchange meetings. MDC also prepared a planning guide detailing in 
user-friendly language the steps to be taken and the tasks to be accomplished in scaling up. 

The report finds that the DEI colleges made demonstrable progress in implementing 
and scaling up developmental education refonns but that they also faced three major challenges. 
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First, the operational definition of “scaling up” was not fully specified, and it changed over 
time. The Request for Proposals called for the colleges to mount strategies that would reach “a 
significant number of students,” and, at the outset, colleges were asked to establish their own 
targets for the number of students to be served by the focal strategies each year. About two- 
thirds of the way through the demonstration, DEI partners and funders introduced a new 
conceptual framework and vocabulary for planning scale-up. They urged the colleges to 
consider how they could move from serving some students in pilot projects to serving more 
students (as they were doing in the DEI) and, finally, to reaching most if not all students eligible 
for a particular intervention. What “most” meant was not defined for individual institutions; it 
was made clear, however, that colleges were not expected to serve most students within the 
demonstration period. 

Second, at the time that the initiative was funded, the limited evidence base then availa- 
ble meant that little was known with certainty about how to improve developmental education 
outcomes for community college students. As a result, most of the focal strategies that the 
colleges proposed to expand and implement represented “best practices” rather than policies 
and programs whose beneficial effects had been proven through rigorous research. 

Finally, the DEI unfolded at a time when community colleges nationwide were expe- 
riencing substantial increases in enrollment. All the DEI institutions registered increased 
student populations, and, at half of them, the number of students rose by 20 percent or more. 
While the proportion of students requiring developmental education rose by just 1 percent 
across all the colleges, this statistic masks considerable variation, with some colleges experi- 
encing double-digit growth in the proportion of students needing remediation. Thus, at the 
same time that the colleges were implementing the new DEI strategies and policies, they were 
facing significant challenges in making instructional and support services available to much 
larger numbers of students. 


The Focal Strategies 

Collectively, the 15 DEI colleges implemented 46 focal strategies. For analytic purposes, these 
strategies were classified as fitting one of four types and as having one of four objectives. The 
great majority (87 percent) of focal strategies were of two types: instructional strategies (chang- 
es in the content of developmental classes or in the means by which they were taught) and 
support strategies (efforts to address students’ academic and personal issues). Nine of the 
colleges adopted at least one of each of these two kinds of strategies. The two remaining types 

of strategies policy changes and strategies directed toward high school students were 

less frequently cited. Moreover, the objectives of these last two kinds of strategies could not 
readily be classified, sometimes because they had multiple objectives. 
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Of the 40 strategies whose objective was identifiable, 19 were classified as aimed at 
providing supports to bolster students’ skills or to help them resolve other problems; examples 
of strategies with this objective include study skills courses, tutoring, and advising. Fourteen 
strategies — including ones involving modularized and computerized courses and paired 
college- level and developmental courses — were aimed at accelerating students’ progress 
through the developmental sequence. Four strategies sought to help students avoid unnecessary 
developmental coursework through placement test preparation, and three entailed implementing 
new approaches to make instruction more relevant and engaging. 


Key Findings About Scaling Up the Focal Strategies 

• Participation in the focal strategies more than doubled over time, but 
most of the strategies did not meet established participation goals. 

Across the colleges, the proportion of incoming developmental students served by at 
least one focal strategy rose from 18 percent in fall 2009 (the initiative’s first semester) to 41 
percent in fall 2011 (the last semester for which data were available). Had participation data 
from spring 2012 been available, the measured increase would have been even larger. Nonethe- 
less, for reasons discussed below, colleges generally did not meet the ambitious participation 
goals that they had set for serving students with their individual focal strategies. Both participa- 
tion data and information about the colleges’ target numbers were available for 33 strategies. 
For four of these strategies, participation equaled or exceeded (sometimes greatly) the targets 
that colleges had established. The majority of strategies fell far short, however, reaching less 
than half the students to whom they were targeted. As noted above, colleges were not expected 
to go from serving “more” to serving “most” students within the time frame of the DEI demon- 
stration. Still, by fall 2011, 10 of the 15 colleges were unable to reach as many as half their 
incoming developmental students with a focal strategy and, thus, had a considerable way to go 
in meeting the longer-term goal of serving most students. Interestingly, there was no difference 
in the extent of scale-up between new strategies and preexisting ones. 

• Resource adequacy, communication, and engagement were three im- 
portant factors promoting large-scale implementation of the focal 
strategies. 

Resources needed for scaling up the strategies included adequate staff, space, and, for 
many interventions, technology. Strong and positive communications helped ensure scale-up, 
with the vocal support of the president proving especially critical. Professional development for 
staff members and staff involvement in planning and oversight committees increased support 
for the strategies. Colleges also learned the importance of marketing the focal strategies to 
students, and they used a variety of media to do so, from brochures to videos to social networks. 
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• A fourth key factor was a departmentwide or institutionwide commit- 
ment to uniform instructional practice; several instructional strategies 
that reflected this commitment were notable because they were begun 
with the intention of serving all students from the outset. 

Three new focal strategies ones involving computerized instruction and the infu- 
sion of content from sociology and history into developmental reading and English courses 

entailed decisions that all faculty members who were teaching sections of a course would teach 
them a certain way. This is a particularly efficient mode of scale-up because all necessary 
faculty resources are already in place and because all students who need developmental classes 
must follow the prescribed approach. In two of the three cases, administrators prescribed the 
new instructional practices in a process referred to here as “gentle fiat”: While instructors may 
help select and may receive professional development on the new approach (this is what makes 
the fiat “gentle”), they are not free to deviate from it. In the third instance, faculty members 
themselves decided to adopt the new approach, and, in general, the experience of the DEI 
institutions indicates that faculty input into the chosen strategy helps to curb resistance to it and 
to make for smoother implementation. 

• In contrast, factors that worked against full scale-up included — along 
with resource limitations — institutional reluctance to impose mandates 
about how students should learn and instructors teach, students’ own 
wishes and priorities, a perceived need to scale back when strategies ap- 
peared to be ineffective, and a desire to evaluate the strategies’ apparent 
effectiveness before moving forward. 

Limited scale-up sometimes reflected colleges’ competing priorities and values. While 
colleges wanted to scale up their focal strategies, some colleges also wanted to give students a 
choice of learning modalities rather than to impose a unifonn approach. Some colleges wanted 
to step back and reevaluate rather than to move forward when an approach did not seem to be 
working. And even when approaches did seem to be working, some colleges wanted some 
evidence of effectiveness before expanding them further. 

Key Findings About Student Outcomes 

The analysis examines five key outcomes: total credits earned in the first tenn, grade point 
average in the first tenn, persistence into the second tenn, passing the “gatekeeper” (first 
college-level English course required for completion of a degree) by the end of the second tenn, 
and passing the gatekeeper math course by the end of the second tenn. 
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A rigorous impact study using random assignment or a strong alternative research de- 
sign was not part of the charge to the evaluators. The methods used in this study can show 
that the DEI was associated with the outcomes that were observed but not that the DEI 
caused these outcomes. The findings below should therefore be regarded as suggestive but in 
no way conclusive. 

• When outcomes for participants in the focal strategies were compared 
with outcomes for nonparticipants, the majority of outcome differ- 
ences (61 percent) were not statistically significant. About a third of the 
strategies were associated with positive gains for students, and a hand- 
ful were associated with negative outcomes. Finally, participation was 
more likely to be associated with positive results for some outcomes 
than for other outcomes. 

One approach to the analysis was to compare outcomes for participants in the strategies 
during their first term with outcomes for students who were eligible but did not participate, 
controlling as much as possible for students’ demographic and achievement-related characteris- 
tics. (Unmeasured differences could not be controlled for, however.) For all but a small number 
of outcomes, participation in the strategies was associated either with better outcomes for 
participants or with no statistically significant differences between outcomes for participants 
and nonparticipants. Participation in the DEI strategies was especially likely to be associated 
with positive results on two of the five outcomes examined: credits earned during the first term 
and passing the gatekeeper English course by the end of the second term. 

• Strategies that involved contextualized instruction and collaborative 
learning were more likely than other kinds of strategies to be associated 
with positive outcome differences. 

Because there are only three examples of such strategies, making generalizations is 
risky; this finding is, however, consistent with the results of other research. Thus, more rigorous 
outside studies confinn that contextualized instruction in vocational programs and learning 
communities has helped students to earn more course credits and progress from developmental 
into college-level coursework. There is also strong evidence supporting the use of structured 
collaborative learning as a pedagogical technique. 

• When outcomes for cohorts of students who enrolled in the colleges be- 
fore the inception of the DEI were compared with outcomes for students 
who enrolled after the DEI was put in place (whether or not these stu- 
dents participated in any of the focal strategies), the later enrollees were 
found to have achieved outcomes that were either better than or si mi lar 
to the outcomes of students who had entered earlier. 
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This approach to examining outcome differences yields findings similar to the approach 
that compares participants with nonparticipants. After the DEI’s implementation, students 
generally did as well as or better than they had previously. Again, factors other than the DEI 
may explain these differences. 

• In general, strategies that reached more than 50 percent of the students 
whom they aimed to serve were more likely to be associated with posi- 
tive outcomes than strategies that reached smaller numbers of students. 

It is not clear whether and to what extent these strategies had to do with the positive 
outcomes. It may be that colleges that were effective in engaging students in their focal strate- 
gies were also likely to be effective more generally. 

From a quantitative point of view, the DEI represents a modest improvement for the 
participating colleges. While the number of participants in the focal strategies doubled over the 
period measured, most strategies fell short both of the numerical targets that colleges had set for 
themselves and of reaching the majority of students in their target populations. And while many 
more strategies were associated with positive outcomes for participants than with negative 
outcomes, the majority of strategies did not make a difference one way or the other. 


Other Outcomes: The DEI in Broader Perspective 

It would be inappropriate to judge the DEI on the basis of the numbers alone, however. To help 
students accelerate through and otherwise succeed in their developmental courses, the colleges 
used DEI funding to support policy changes and programmatic reforms beyond those included 
in their focal strategies. The colleges also used their DEI grants to support on-campus profes- 
sional development on a broad range of topics related to developmental education, including the 
use of new instructional modalities, the characteristics and needs of low-income students, and 
how instructors could help students better meet those needs. College personnel learned from 
one another and from their counterparts at non-DEI colleges at conferences and meetings whose 
attendance was made possible by DEI monies. And, at some colleges, the DEI stimulated wider 
discussions about student success and campus priorities. 

It also appears that the DEI will leave a lasting legacy at participating colleges. At 
some institutions, leaders were committed from the start to continuing to support the focal 
strategies with regular college funding after the DEI grants expired; at other institutions, 
leaders did not make such advance commitments but have opted to move forward with the 
strategies that they deemed successful. Some DEI innovations will carry over into future 
initiatives — notably, the Bill & Melinda Gates Foundation’s new community college reform 
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effort known as Completion by Design. And the intervention has brought forward a new group 
of able and thoughtful administrators. 

At the outset of the DEI, little was known about what is required for scaling up initia- 
tives in community colleges. This study suggests that additional resources may be necessary but 
are not sufficient. Also critical are communication, engagement, and a commitment to uniform 
practice throughout a department or institution. Time is also critical, not just for putting new 
interventions in place but also for securing the buy-in and support needed for smooth imple- 
mentation. Yet another lesson concerns the importance of having expectations that are well 
specified and shared by all parties. Finally, the experiences of the DEI colleges serve as remind- 
ers that scale-up is just one of many objectives that community colleges strive to meet, that the 
complexities of students’ lives can interfere with scale-up efforts, and that both high ambitions 
and realistic expectations for expanding promising initiatives are in order. 
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Chapter 1 

Introduction 


Postsecondary education has increasingly become a prerequisite for entry into the middle class. 
This has become even more true as the United States emerges from the Great Recession. Better- 
paying jobs demand credentials beyond a high school diploma. But as a 2010 report by the 
Georgetown University Center on Education and the Workforce indicates, the country is 
producing too few postsecondary degrees and certificates. As employers demand higher levels 
of education and training, this shortfall — of an estimated 3 million associate’s degrees or 
higher, along with 4.7 million postsecondary certificates — will consign those who have lesser 
credentials to lower-paying jobs or to no jobs at all. 1 Community colleges, already the largest 
component of the nation’s postsecondary system, have been called on to help fill this gap. 2 

Community colleges play a particularly important role in serving low-income students 
and older adults who have been away from the classroom for many years. These students are 
especially likely — based on assessment tests administered upon enrollment — to be deemed to 
need remedial (“developmental”) classes in reading, composition, and/or mathematics before 
embarking on college-level coursework that counts toward graduation. Research indicates that 
over half of all community college students are judged to need at least one developmental class. 
Developmental classes themselves, however, have all too often proved not just the starting point 
but also the end to students’ progress through community college. The majority of students who 
are referred to developmental education do not complete their prescribed sequence of remedial 
courses, much less persevere to obtain a diploma or certificate. 3 

Recognizing the significance of developmental education as a stumbling block in the 
path to college graduation, the Bill & Melinda Gates Foundation created the Developmental 
Education Initiative (DEI) in 2009, and Lumina Foundation for Education funded the evalua- 
tion. Fifteen colleges in six states, shown in Table 1.1, each received a grant of $743,000 over a 
three-year period. All the colleges were early participants in Achieving the Dream (AtD): 
Community Colleges Count, a national community college reform network dedicated to 
evidence-based decision-making. MDC, Inc. — a North Carolina-based nonprofit organization 
that had been the managing partner of AtD until Achieving the Dream, Inc., was established as 
a separate entity in 2010 — was selected as managing partner of the DEI as well. 


'Carnevale, Smith, and Strohl (2010). 

2 White House Summit on Community Colleges (2010). 

3 Jenkins, Jaggars, and Roksa (2009); Bailey, Jeong, and Cho (2010). 
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The Developmental Education Initiative 
Table 1.1 

States and Colleges Participating in the Developmental Education Initiative 


Connecticut 

• Housatonic Community College 

• Norwalk Community College 


Florida 

• Valencia College 


North Carolina 

• Guilford Technical Community College 


Ohio 

• Cuyahoga Community College 

• Eastern Gateway Community College 

• North Central State College 

• Sinclair Community College 

• Zane State College 

Texas 

• Coastal Bend College 

• El Paso Community College 

• Houston Community College 

• South Texas College 

Virginia 

• Danville Community College 

• Patrick Henry Community College 


The purpose of the DEI grants to the colleges was to enable the institutions to establish 
and scale up interventions that would help students to progress through developmental courses 
more rapidly and more successfully or to bypass these courses altogether. DEI funding also 
financed state policy teams that have sought to influence state higher education legislation and 
policies to enhance student success. In its reliance on both institutional change at the community 
college and supportive state policy work, the DEI follows the AtD model and theory of change. 

Seven organizations, shown in Table 1.2, make up the partnership that provides lead- 
ership and support for the DEL MDC worked closely with the Bill & Melinda Gates Founda- 
tion to design the initiative and has directed the entire effort, providing guidance to the 
colleges, organizing communications, overseeing compliance, reviewing annual report 
documents, and convening regular meetings of the colleges and other parties. 4 Jobs for the 
Future has helped the state policy teams to develop a policy agenda, engage the colleges 
around this agenda, and generate support for identified reforms. The Community College 
Leadership Program (CCLP) at the University of Texas at Austin has organized and deployed 
a cadre of consultants to provide technical assistance to the colleges as the colleges deemed 


4 MDC also published a detailed planning guide (Parcell, 2012), discussed below. 
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The Developmental Education Initiative 
Table 1.2 


Organizational Partners in the Developmental Education Initiative 


Organization 

Primary Responsibilities 

American Association of Community 
Colleges 

http://www.aacc.nche.edu 

Provides DEI a voice in national conversations 
about issues and policies that affect developmental 
education students through its annual conferences 
and meetings and publications; gives DEI colleges 
access to data analysis tools through the Achieving 
the Dream national database 

Community College Leadership Program 
University of Texas at Austin 
http://utcclp.org 

Trains and deploys a cadre of technical assistance 
providers to support the work of DEI colleges; 
collaborates with other partners on the design and 
delivery of DEI learning events 

Community College Research Center 
Teachers College, Columbia University 
http://ccrc.tc.columbia.edu 

Assists MDRC with the evaluation, conducting the 
quantitative analysis of data submitted by the 
colleges 

Jobs for the Future 
http://www.jff.org 

Provides technical assistance to DEI state policy 
teams and supports a learning network among 
states, including semiannual learning events 

MDC 

http://www.mdcinc.org 

Directs the overall initiative, including compliance 
and communications, integrating lessons emerging 
from the participating colleges and states, 
coordinating partners in the design and delivery of 
that learning, and developing tools and resources 
for scaling and sustaining innovations 

MDRC 

http://www.mdrc.org 

Leads the evaluation of the initiative 

Public Agenda 
http://www.publicagenda.org 

Augments the communications and engagement 
strategy of the initiative, conducting focus groups 
with college administrators and business leaders on 
the topic of developmental education 


necessary . 5 MDRC, a nonprofit, nonpartisan policy research organization, is the principal 
evaluator of the DEI, with the Community College Research Center (CCRC) at Teachers Col- 


5 The DEI colleges could also contract for technical assistance from individuals outside the CCLP pool. 
The colleges had additional opportunities for expert consultation as part of their participation in AtD. (Each 
AtD institution receives one visit each year from its AtD leadership coach and data coach.) AtD coaching 

(continued) 
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lege, Columbia University, serving as evaluation partner and focusing on quantitative outcomes. 
The American Association of Community Colleges and Public Agenda have played subsidiary 
roles. 6 

The directive to the evaluators was to examine the implementation of the DEI at the 
participating colleges; the state policy work, while an important component of the initiative, is 
outside the scope of the research effort. A May 201 1 evaluation report considered the initia- 
tive’s evolution during the first year of the grant; its objective was to explain the elements that 
facilitated or constrained early progress in implementing strategies that the 15 schools designat- 
ed as their key, or “focal,” strategies. 7 

This is the second and final MDRC report on the initiative. It addresses three main 
questions: 

• To what extent did the colleges scale up their chosen developmental educa- 
tion reforms to serve more students? 

• What factors affected the colleges’ ability to expand their programs and prac- 
tices? 

• To what extent were the colleges’ strategies associated with improvements in 
student outcomes? 8 

The report also considers ways that participation in the DEI influenced the colleges more 
broadly. For these reasons, the report may be of interest to other colleges looking to scale up 
reforms (especially reforms that are related to instruction and the provision of student supports), 
as well as to funders concerned about how best to support community colleges in bringing 
promising ideas to scale. 

The remaining sections of this chapter provide background information about the initia- 
tive and the evaluation, discuss the methodology on which the report is based, and preview the 
remaining chapters. 

teams were instructed to focus their 2010-201 1 and 2011-2012 visits to these colleges on the DEI interventions, 
on scaling, and on the related work of institutional change. 

6 ln addition. Achieving the Dream, Inc., which was established after the inception of the DEI, helped to 
share findings from DEI colleges throughout the larger AtD national reform network and provided a platform 
for DEI institutions to present at the AtD Annual Convening on Student Success and through national 
conference calls. 

7 See Quint et al. (2011). 

8 As discussed below, the evaluation was not funded or designed to determine whether or not the strategies 
actually caused these outcomes. Thus, the report can provide preliminary but not definitive evidence about the 
effectiveness of the interventions. 
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The Background of the Initiative and of This Study 

As noted above, the DEI is an outgrowth of Achieving the Dream. As part of that initiative, a 
number of the colleges had developed strategies to help students to progress through develop- 
mental education more quickly or to sidestep developmental coursework entirely and had begun 
to review evidence on the effectiveness of these interventions. The foundations’ original goal 
was for the colleges to expand those AtD strategies that, according to the colleges’ internal 
evaluations, had demonstrated success. As a result of early discussions, however, the funders 
agreed that colleges should be allowed to develop new initiatives as well as to scale up existing 
interventions. 

At the inception of the demonstration, previous efforts to scale up interventions at 
community colleges had largely gone unexamined, and blueprints and guidelines were unavail- 
able. Colleges were encouraged to explore new avenues for expanding what they were doing, 
with the idea that they could make midcourse corrections if necessary. Learning what it takes to 
“grow” interventions was a major goal. An initial assumption was that limited financial re- 
sources were a major stumbling block to bringing promising interventions to scale, and grant 
levels were set with this assumption in mind. 9 Modest funding was set aside for colleges to 
receive some coaching from experts (including former college presidents) who had experience 
with developmental education reforms and bringing programs to scale. 

The proposals that the colleges submitted were extremely ambitious — the number of 
strategies they proposed to implement or expand ranged from 3 to 14 — and typically included 
a mix of existing and new interventions. At the DEI Kickoff Institute in July 2009, the colleges 
were urged to take on fewer strategies, but to do these well. They were also asked to identify 
up to three “focal” strategies that could be expected to serve increasing numbers of students 
over time, and it was agreed that the evaluation would center on these strategies, tracking their 
scaling up both qualitatively and quantitatively. In general, the colleges stuck with these focal 
strategies over the course of the three-year demonstration. In some cases, however, they added 
new strategies to the mix. And one focal strategy that was agreed upon early on was never 
implemented. 

Within the first year, it became evident that the initial assumptions about what it would 
take for colleges to achieve scale were overly optimistic and that many colleges needed more 
assistance in planning their scaling efforts. MDC added coaching visits to support the colleges 
and convened additional meetings to support peer learning. MDC also began to prepare a 
planning guide, the first of its kind directed toward community colleges. Entitled More to Most: 


9 ln comparison, colleges participating in the first round of AtD received a planning grant of $50,000 and 
grants of $ 1 00,000 during each of the first four years of implementation. 
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Scaling Up Effective Community College Practices, the guide details steps to be taken and tasks 
to be accomplished in going to scale. 10 

Three points bear additional mention. First, the goals for expansion were initially un- 
clear and changed two-thirds of the way through the demonstration. While the Request for 
Proposals called for the colleges to mount strategies that would reach a “significant number of 
students who need developmental education with a goal of major performance improvement,” 
just what constituted a “significant number” was largely left undefined. After the Kickoff 
Institute, colleges were asked to estimate the number of students that each of their focal strate- 
gies would reach each year, and, during the first two years, all parties accepted these estimates 
as the participation targets. In the last year of the demonstration, the funder asked that colleges 
be introduced to a new framework and a new ultimate objective: to move from “more to most” 
students eligible for the intervention, although colleges were not expected to attain this goal 
within the grant period. Again, the meaning of “most” — whether it signified a simple majority 
or a figure much closer to 100 percent — was not specified. 11 

Second, when the initiative was funded, much was speculated but little was known with 
certainty about what works to improve outcomes for community college students. Most of the 
strategies that the colleges proposed and subsequently pursued represent “promising practices” 
— that is, policies and programs that made intuitive sense, that had often been tried previously 
at the colleges themselves or at other institutions, and that sometimes had been associated with 
improved student outcomes. These policies and programs had not, however, been subjected to 
rigorous evaluation, and, for most, there was no strong evidence that they had actually caused 
the better results for students. Thus, the funders’ hopes and expectations that expanding the 
reach of previously tried interventions would enable more students to obtain a postsecondary 
credential in a timely manner exceeded the limited knowledge base about what works to 
improve outcomes for community college students. 12 

Finally, the DEI unfolded at a time when community colleges across the country were 
experiencing substantial increases in enrollment, and the DEI colleges were no exception. 
Between fall 2008 and fall 2010, total enrollment grew at all 15 institutions, and, at half the 
colleges, the number of students rose by 20 percent or more during the two-year period. Thus, at 
the same time that the colleges were implementing the new initiative, they were experiencing 
significant challenges in making instructional and support services available to much larger 


10 Parcell (2012). 

n For interventions that were narrowly targeted (for example, to the highest-level developmental students), 
moving from more to most would not necessarily mean reaching a large number of students eligible for the 
intervention, nor a large share of all students requiring developmental coursework. 

12 A recent MDRC report summarizes what is known about the effectiveness of various strategies aimed at 
enhancing outcomes for developmental students. See Zachry Rutschow and Schneider (2011). 
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student populations. The academic preparation of the new students did not differ greatly from 
that of previous cohorts; on average across the schools, the percentage of new students referred 
to developmental education increased by just one percentage point between 2007-2008 and 
2010-2011. This statistic masks considerable variation: Some colleges experienced double-digit 
growth in the proportion of students needing remediation, while other colleges saw this propor- 
tion decrease over time. Given the overall enrollment boom, however, DEI schools experienced 
a sizable increase in developmental enrollments. 


The Methodology of the Report 

This report relies on a combination of qualitative and quantitative data. Throughout the three- 
year study period, MDRC researchers conducted telephone interviews each semester with the 
DEI liaisons at all the colleges to leam about the progress of their efforts to implement and scale 
up their colleges’ focal strategies. The researchers conducted two-day site visits to the colleges 
in spring 2010 to leam more about the colleges and their implementation experiences at first 
hand. During these visits, they met with college presidents and other leading administrators, as 
well as with staff members charged with putting the focal strategies into practice. 

On the basis of their promising early experiences, MDRC selected five colleges — 
Danville Community College, El Paso Community College, Guilford Technical Community 
College, Sinclair Community College, and Valencia College — as the subjects of case studies, 
which were conducted in fall and winter of the 2011-2012 academic year. Again, researchers 
interviewed key administrators and faculty members but this time were able to include observa- 
tions of classes and focus groups with students among their activities while on-site. 

The decision to center most of the data collection on the focal strategies was both a 
strength and a limitation of the study. It allowed for a more in-depth understanding of the 
implementation of these strategies. But it also meant that other important developments at the 
colleges that were not included among the focal strategies (such as policy changes that were 
implemented all at once rather than gradually) may have received less attention than they 
warranted. As a partial corrective, the researchers reviewed the annual reports that colleges 
submitted to MDC, which were broad in their scope. And the final interview that the researchers 
conducted with the DEI liaisons in spring 2012 asked the liaisons to reflect more expansively on 
what the DEI had meant for their institutions. 

On the quantitative front, colleges were instructed to submit data on participation in 
the focal strategies to JBL Associates (JBL), which processed these data and forwarded the 
data files to CCRC for analysis. Colleges were also asked to identify comparison groups of 
students who were eligible for the interventions but did not participate in them. This infor- 
mation allows for a quantitative analysis of the extent to which the strategies were scaled up 
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over time. It also permits an examination of the degree to which changes in outcomes were 
associated with participation in a particular strategy. The evaluation was not designed to assess 
the effects of specific strategies through random assignment of students to program and control 
groups. (The number of colleges and strategies involved in the DEI would have made this a 
daunting proposition.) Without random assigmnent, however, it is impossible to know just how 
closely nonparticipants in the colleges’ comparison group resembled the participants before- 
hand, so it is equally impossible to be certain that participation (rather than preexisting differ- 
ences between participants and nonparticipants) caused the change in outcomes. While this 
report takes a sizable step beyond the colleges’ own self-reports, it is not able to answer 
definitively whether the interventions made a difference for students or which interventions 
were more effective in this regard. 

The DEI colleges were encouraged to take risks and to share lessons — good and bad 
— with other colleges. They were not asked to compete with one another. For this reason, in 
tables presenting the scaling and outcome findings, the 1 5 colleges are indicated simply by the 
letters A to O and are not shown in any particular order. In the qualitative analyses, colleges are 
usually discussed by name because they have important lessons to teach. 


The Organization of the Report 

The remainder of the report contains five chapters. After this introductory chapter, 

• Chapter 2 describes the participating colleges and reviews the focal DEI 
strategies and their characteristics. 

• Chapter 3 is the first of two chapters that deal with the implementation of the 
initiative. It examines scale-up from both quantitative and qualitative per- 
spectives, drawing on the data submitted to JBL as well as on infonnation 
collected during the case study site visits, other site visits, and telephone in- 
terviews, to understand the extent to which colleges expanded their focal 
strategies and the forces that promoted or hindered that expansion. 

• Chapter 4 moves beyond the focal strategies to explore other ways in which 
the DEI influenced the participating colleges. 

• Chapter 5 considers the extent to which the specific DEI focal strategies and 
the initiative as a whole are associated with improved student outcomes. 

• Chapter 6 reflects on the DEI’s lessons for funders, policymakers, and 
practitioners. 



Chapter 2 

The DEI Colleges and Their Focal Strategies 


This chapter presents a further backdrop for the rest of the report. It describes the 15 
colleges that participated in the Developmental Education Initiative (DEI). It also discusses 
the characteristics of the focal strategies that colleges put in place. 


Key Findings 

• The colleges were highly varied in their characteristics and in the characteris- 
tics of their students. 

• The great majority of the focal strategies entailed either instructional refonns 
or student supports. 

• The focal strategies were almost evenly divided between ones that were new 
and ones that were scaled up from existing interventions. 

• The majority of focal strategies aimed to help students move more quickly 
through developmental education or to provide them with academic and/or 
personal support. 


The Participating Colleges 

As early participants in the Achieving the Dream (AtD) national refonn network, the colleges 
participating in the DEI had expressed a strong commitment to using data to improve programs 
and achieve better outcomes for their students. Aside from this common trait, the colleges differ 
in their institutional characteristics and in the characteristics of their students, as Tables 2.1a and 
2.1b make clear. 

Three of the colleges are located in large cities (Cleveland, El Paso, and Houston); four 
are in small towns or rural areas; and the remainder are in small- to medium-size cities or their 
suburbs. In fall 2010 (the most recent year for which data are available), six colleges enrolled 
fewer than 5,000 students, and six had 20,000 students or more, with the Houston Community 
College System serving more than 60,000 students. 

Women and students under age 25 constituted the majority across the colleges. The ma- 
jority of colleges also enrolled more part-time than full-time students. Seven colleges predomi- 
nantly served students of color, primarily black and Hispanic students; at three additional 
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The Developmental Education Initiative 
Table 2.1a 


Characteristics of Colleges Participating in the Developmental Education Initiative, Part I 



Coastal 

Bend 

College 

Cuyahoga 

Community 

College 

District 

Danville 

Community 

College 

Eastern 

Gateway 

Community 

College 

El Paso 
Community 
College 

Guilford 

Technical 

Community 

College 

Housatonic 

Community 

College 

Houston 

Community 

College 

Location 

Beeville, 

Cleveland, 

Danville, 

Steubenville, 

El Paso, 

Jamestown, 

Bridgeport, 

Houston, 


TX 

OH 

VA 

OH 

TX 

NC 

CT 

TX 

Degree of urbanization 

Small 

town/rural 

Large city 

Small city 

Small city 

Large city 

Suburb of 
midsize city 

Midsize city 

Large city 

Number of campuses 

4 

3 a 

1 

1 

5 

4 

1 

18 

Published in-district tuition 

2,400 

2,537 

3,257 

2,790 

1,690 

1,981 

3,406 

1,394 

and fees' 3 ($) 

Fall 2010 enrollment 

Total enrollment 

4,348 

31,250 

4,534 

2,209 

29,909 

14,789 

6,197 

60,303 

Full-time students (%) 

40 

41 

38 

55 

39 

61 

38 

31 

Part-time students (%) 

60 

59 

62 

45 

61 

39 

62 

69 

Male (%) 

39 

38 

41 

40 

43 

44 

38 

41 

Female (%) 

61 

62 

59 

60 

57 

56 

62 

59 

Black, non-Hispanic (%) 

3 

33 

40 

11 

2 

44 

27 

29 

White, non-Hispanic (%) 

29 

54 

57 

75 

9 

44 

37 

18 

Hispanic (%) 

65 

4 

2 

1 

85 

4 

24 

31 

American Indian, Alaska Native, 

1 

2 

1 

2 

1 

4 

3 

10 

Asian, or Pacific Islander (%) 

Race/ethnicity unknown (%) 

2 

5 

1 

11 

<1 

3 

6 

1 

Foreign/nonresident c (%) 

<1 

2 

<1 

<1 

2 

1 

1 

10 

Two or more races (%) 

<1 

<1 

<1 

1 

<1 

1 

1 

1 

Linder age 25 (%) 

NA 

50 

65 

59 

68 

54 

59 

55 
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Table 2.1a (continued) 



North 

Central 

State 

College 

Norwalk 

Community 

College 

Patrick 

Henry 

Community 

College 

Sinclair 

Community 

College 

South 

Texas 

College 

Valencia 

College 

Zane 

State 

College 

Average 

(unweighted) 

Location 

Mansfield, 

OH 

Norwalk, 

CT 

Martinsville, 

VA 

Dayton, 

OH 

McAllen, 

TX 

Orlando, 

FL 

Zanesville, 

OH 


Degree of urbanization 

Small 

town/rural 

Small city 

Small 

town/rural 

Midsize city 

Midsize city 

Midsize city 

Small 

town/rural 


Number of campuses 

2 

1 

1 

l a 

5 

8 

1 


Published in-district tuition 

3,114 

3,386 

3,280 

1,735 

2,364 

2,752 

4,130 

2,681 

and fees 6 ($) 

Fall 2010 enrollment 

Total enrollment 

3,635 

6,740 

3,289 

21,994 

27,692 

41,583 

2,857 

17,422 

Full-time students (%) 

39 

38 

50 

46 

33 

43 

68 

44 

Part-time students (%) 

61 

62 

50 

54 

67 

57 

32 

56 

Male (%) 

37 

41 

39 

43 

43 

44 

42 

41 

Female (%) 

63 

59 

61 

57 

57 

56 

58 

59 

Black, non-Hispanic (%) 

6 

16 

25 

15 

<1 

17 

5 

18 

White, non-Hispanic (%) 

86 

40 

71 

67 

1 

38 

90 

48 

Hispanic (%) 

1 

24 

2 

1 

87 

29 

1 

24 

American Indian, Alaska Native, 

1 

4 

1 

2 

<1 

5 

1 

3 

Asian, or Pacific Islander (%) 

Race/ethnicity unknown (%) 

5 

11 

1 

14 

9 

9 

2 

5 

Foreign/nonresident 0 (%) 

<1 

3 

<1 

1 

3 

2 

0 

2 

Two or more races (%) 

1 

2 

1 

1 

<1 

1 

1 

1 

Under age 25 (%) 

54 

59 

56 

49 

74 

68 

50 

59 


SOURCES: U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary Education Data System (IPEDS), and the colleges' 
Web sites. 


NOTES: NA = not available. Rounding may cause slight discrepancies in sums and differences. 

“College has added one campus since the start of the grant. 

b Student expense data are for full-time, first-time, degree- or certificate-seeking students in the 2010-2011 school year. 
c Raee/ethnicity is unknown. 





The Developmental Education Initiative 
Table 2.1b 


Characteristics of Colleges Participating in the Developmental Education Initiative, Part II 



Coastal 

Bend 

College 

Cuyahoga 

Community 

College 

District 

Danville 

Community 

College 

Eastern 

Gateway 

Community 

College 

El Paso 
Community 
College 

Guilford 

Technical 

Community 

College 

Housatonic 

Community 

College 

Houston 

Community 

College 

System 

Developmental education referral 3 (%) 

Students referred to developmental math 

32 

70 

66 

31 

69 

61 

53 

58 

Students referred to developmental English 

27 

40 

27 

22 

31 

35 

63 

22 

Students referred to developmental reading 

25 

1 

7 

9 

41 

32 

64 

18 

Students referred to any developmental education 

34 

72 

70 

40 

73 

67 

69 

60 

Financial aid b 

Any financial aid received (%) 

85 

66 

79 

84 

81 

74 

66 

57 

Institutional grant aid received (%) 

12 

2 

15 

24 

3 

5 

26 

3 

Average institutional grant aid ($) 

Retention and completion 

879 

990 

967 

2,052 

1,457 

1,379 

1,447 

625 

Retention rate for first-time students 0 

Full-time students (%) 

45 

51 

64 

56 

70 

56 

63 

61 

Part-time students (%) 

40 

36 

50 

31 

52 

43 

54 

49 

Graduation rate d (%) 

21 

4 

23 

23 

11 

13 

8 

12 

Transfer-out rate 0 (%) 

19 

10 

11 

15 

21 

23 

20 

27 
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Table 2.1b (continued) 



North 

Central 

State 

College 

Norwalk 

Community 

College 

Patrick 

Henry 

Community 

College 

Sinclair 

Community 

College 

South 

Texas 

College 

Valencia 

College 

Zane 

State 

College 

Average 

(unweighted) 

Developmental education referral 3 (%) 

Students referred to developmental math 

57 

NA 

73 

55 

38 

48 

61 

55 

Students referred to developmental English 

40 

NA 

50 

45 

40 

29 

53 

37 

Students referred to developmental reading 

21 

NA 

42 

30 

38 

36 

46 

29 

Students referred to any developmental education 

65 

NA 

78 

64 

62 

59 

75 

63 

Financial aid 1 ’ 

Any financial aid received (%) 

87 

61 

83 

75 

90 

84 

98 

78 

Institutional grant aid received (%) 

9 

27 

0 

8 

0 

15 

16 

11 

Average institutional grant aid ($) 

Retention and completion 

1,076 

1,728 

1,018 

992 

1,459 

1,271 

2,424 

1,318 

Retention rate for first-time students'" 

Full-time students (%) 

47 

66 

52 

54 

60 

70 

56 

58 

Part-time students (%) 

43 

42 

39 

47 

50 

56 

45 

45 

Graduation rate d (%) 

13 

8 

27 

8 

13 

40 

23 

16 

Transfer-out rate 3 (%) 

NA 

21 

11 

20 

10 

13 

NA 

17 


SOURCES: U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary Education Data System (IPEDS), and colleges’ 
reports to JBL. 

NOTES: NA = not available. Rounding may cause slight discrepancies in sums and differences. 

“Developmental education referral rates measure the percentage of an incoming student cohort who are referred to developmental math, English, or reading, 
respectively. Rates are calculated as an average from three academic years: 2008-2009, 2009-2010, and 2010-201 1 . 

b Financial aid data refer to full-time, first-time, degree- or certificate-seeking students for the 2009-2010 academic year. 

c The retention rate measures the percentages of first-time students who began their studies in fall 2009 and who returned to the institution in fall 2010. 
d Graduation rates are for the cohort of full-time, first-time, degree- or certificate-seeking students who began their studies in fall 2007. They measure the 
percentage of these students who complete a degree or certificate within 150 percent of the normal time to program completion. 

c Transfer-out rates measure the percentage of full-time, first-time students entering in fall 2007 who are known to have transferred to another institution within 
150 percent of the normal time to program completion. 




schools, between one-third and one-half the students belonged to one of these two groups. At all 
the colleges, the majority of students received financial aid. 

Across all the colleges, an average of 55 percent of students were referred to develop- 
mental mathematics courses; 29 percent were referred to developmental reading courses; and 37 
percent were referred to developmental writing courses (often referred to as “developmental 
English” and, in some instances, integrated with developmental reading). Nontraditional and 
academically underprepared students typically have high dropout rates. Some 53 percent of full- 
time students and 36 percent of part-time students who enrolled in the DEI colleges for the first 
time in fall 2007 enrolled for a second year, with the rest interrupting their academic careers at 
the college or dropping out altogether. Some 16 percent of the first-time, full-time, degree- or 
certificate-seeking students who entered in fall 2007 received a degree or certificate within 150 
percent of the minimum time necessary for program completion, while another 17 percent of 
these students transferred to another institution within this time period. 


The Focal Strategies 

Collectively, the 15 community colleges that participated in the DEI implemented 46 focal 
strategies. Table 2.2 provides an overview, presenting the focal strategies that each college 
adopted, along with three salient attributes: the strategy’s type, whether it was new or scaled 
up, and — especially important for this report — the strategy’s objective. Each of these is 
discussed below. 

Strategy Type 

Table 2.2 classifies each strategy as fitting into one of four broad types. 

• Policy strategies are designed to change collegewide policies and practices. 

These policies commonly relate to placement, registration, enrollment, and 
course requirements or sequencing. 

• Instructional strategies entail changes in the content of developmental 
courses and the modality by which these courses are taught. They also in- 
clude two or more courses linked into “learning communities,” ideally with 
mutually reinforcing themes and assignments. 

• Support strategies are efforts to address both academic and personal issues 
that students may confront. Such strategies range from advising to tutoring. 
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The Developmental Education Initiative 

Table 2.2 

Focal Strategies of the DEI Colleges 


College 

Strategy 

Strategy 

Type 

New or 
Scaled Up b 

Strategy 

Objective 11 

Coastal Bend 
College 

• Mandate for continuous 
enrollment in developmental 
sequence 

Policy 

New 

NA 


• Case management 

Support 

New 

Support 


• Compression of developmental 
and college-level English 

Instructional 

New 

Acceleration 


• Fast Track (short, intensive 
English and reading courses) 

Instructional 

New 

Acceleration 

Cuyahoga 

Community 

• Supplemental instruction in 
English and math 

Support 

Scaled up 

Support 

College 


• Pairing of Math 950 with Math 
850 (student support course) 

Instructional 

Scaled up 

Support 


• Mentoring for students in 
paired math strategy and for 
new students on Metro campus 

Support 

Scaled up 

Support 

Danville 

Community 

• Modularization in math 
through MyMathLab 

Instructional 

New 

Accleration 

College 


• Blocking late registration for 
students with developmental 
placements in math, reading, 
and writing 

Policy 

New 

NA 


• Preparation for Compass 
placement test 

Support 

New 

Avoidance 

Eastern 

Gateway 

Community 

College 

• Mandate to complete 
developmental requirements 
before proceeding to college- 
level classes 

Policy 

New 

NA 


• Redesign of developmental math 

Instructional 

New 

Acceleration 


• Redesign of developmental 
English 

Instructional 

New 

Acceleration 


• Integration of low-level 
developmental education with 
ABLE programs 

Instructional 

New 

NA 


(continued) 
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Table 2.2 (continued) 


College 

Strategy 

Strategy 

Type 

New or 
Scaled Up b 

Strategy 

Objective" 

El Paso 

Community 

College 

• Pretesting Retesting Education 
Program (PREP) - placement 
test orientation 

Support 

Scaled up 

Avoidance 


• Self-paced, computerized math 
through Math Emporium 

Instructional 

New 

Acceleration 


• Case management 

Support 

New 

Support 

Guilford 

Technical 

Community 

College 

• Specialization and extension of 
SOAR (Student Orientation, 
Advising, and Registration) 
program for students placing 
into two or more areas of 
developmental education 

Support 

Scaled up d 

Support 


• Online or in-person review for 
college placement test 

Support 

New 

Avoidance 


• Light-touch advocacy for all 
students; intensive case 
management for students 
placing into two or more areas 
of developmental education 

Support 

New 

Support 

Housatonic 

Community 

• Modularized, open-entry/open- 
exit (OE/OE) English 

Instructional 

New 

Acceleration 

College 


• Modularized, OE/OE algebra 

Instructional 

Scaled up 

Acceleration 


• Modularized, OE/OE pre-algebra 

Instructional 

Scaled up 

Acceleration 

Houston 

Community 

College 

• Mandated freshman success/ 
Guided Studies (GUST) course 
for all first-time students 

Support 

Scaled up 

Support 

System 


• Learning community linking 
GUST to developmental math 
and English 

Instructional 

Scaled up 

Support 


• Eight-week math bridge courses 
to help students on the cusp 
move up a developmental 
course level 

Support 

Scaled up 

Acceleration 


(continued) 
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Table 2.2 (continued) 


College 

Strategy 

Strategy 

FT* 3 

Type 

New or 
Scaled Up b 

Strategy 

Objective" 

North 
Central 
State College 

• Redesign of assessment and 
placement, including cut-points 
for developmental placement 

Policy 

New 

NA 


• One-week fast-track math 
boot camp 

Support 

Scaled up 

Avoidance 


• Expansion of tutoring - writing 

Support 

Scaled up 

Support 


• Expansion of tutoring - math 

Support 

Scaled up 

Support 

Norwalk 

Community 

College 

• Learning community pairing 
upper-level developmental 
writing with a student success 
course 

Instructional 

Scaled up 

Support 

Patrick 

Henry 

Community 

• Active/cooperative learning 
pedagogy 

Instructional 

Scaled up 

Instructional 

relevance 

College 

• Enhanced advising (creation of 
student database to identify 
high-risk students and enhance 
"continuity of care" across 
advising staff) 

Support 

Scaled up 

Support 


• Fast Track: Accelerated Learning 
Program (ALP) pairing highest- 
level developmental English 
with college-level English 

Instructional 

Scaled up 

Acceleration 

Sinclair 

Community 

College 

• Early Support Program (case 
management in eight high school 
"college and career centers") 

High school 

New e 

NA 


• Developmental math modules 
and boot camp 

Instructional 

New 

Acceleration 


• Accelerated Learning Program 
(ALP) pairing highest-level 
developmental English with 
college-level English 

Instructional 

New 

Acceleration 


(continued) 
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Table 2.2 (continued) 


College 

Strategy 

Strategy 

Type 

New or 
Scaled Up b 

Strategy 

Objective" 

South 

Texas 

College 

• Contextualization of 
developmental reading and 
English curricula 

Instructional 

New 

Instructional 

relevance 


• Case management for 
developmental reading and 
English students (face-to-face) 

Support 

Scaled up 

Support 


• Case management for 
developmental reading and 
English students (e-mail/phone) 

Support 

Scaled up 

Support 

Valencia 

College 

• Learning communities pairing 
developmental courses with 
student success course' 

Instructional 

Scaled up 

Support 


• Supplemental Learning Leaders 
in the classrooms' 

Support 

Scaled up 

Support 


• High school bridge program 
(scholarships and intensive 
supports for 250 high-risk, low- 
income high school students) 

High school 

Scaled up 

NA 

Zane 

State 

College 

• Paired and compressed 
developmental math course 

Instructional 

New 

Acceleration 


• Pairing developmental reading 
or English with college-level 
courses 

Instructional 

New 

Instructional 

relevance 


• Scaled-up advising for 
developmental reading and 
English students 

Support 

Scaled up 

Support 


(continued) 
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Table 2.2 (continued) 


NOTES: NA = not applicable. 

Strategies are categorized into four broad types: (1) "policy" strategies are those designed to 
change institutionwide policies and practices around placement, registration, enrollment, and course 
requirements/sequencing; (2) "support" strategies are those designed to improve academic and student 
service supports beyond the traditional classroom; (3) "instructional" strategies are those designed to 
reach students in the classroom through changes in curriculum and instruction; and (4) "high school" 
strategies are those focused on precollege interventions. 

b At a given college, a strategy is categorized as "new" if it had not been implemented at the college 
prior to the DEI and as "scaled up" if it was expanded from a preexisting intervention. 

c The four strategy objectives — acceleration, avoidance, instructional relevance, and support — are 
drawn from Unlocking the Gate: What We Know About Improving Developmental Education (Zachry 
Rutschow and Schneider, 2011). The authors offer the following definitions: avoidance strategies help 
students avoid developmental education by shoring up their skills before they enter college; 
acceleration strategies are interventions that accelerate students’ progress through developmental 
education by shortening the timing or content of their courses; contextualization strategies are 
programs that provide basic skills together with occupational or college-content coursework (including 
learning communities); and support strategies are programs that enhance the supports for 
developmental-level learners, such as advising or tutoring. For the purposes of this report, 
"contextualization" has been broadened into "instructional relevance" to better encompass the colleges' 
strategies in this category. 

d This strategy was implemented at Guilford Tech under Achieving the Dream but has been refined 
for a narrower group of students (those with two or more developmental education placements). 

c This strategy was previously in place for Sinclair students but is being brought into the high 
schools under the DEL It is categorized as "new" due to the unique challenges of working with new 
institutional partners and a different pool of students. 
f This is not supported by DEI fiinding. 


• High school strategies assist high school students (who may or may not 
subsequently enroll in the college), often with comprehensive services. 1 

Figure 2.1 illustrates the distribution of the focal strategies by type. The great majority 
of strategies fall under two strategy types — instructional reforms and student supports — 
which account, respectively, for 46 percent and 41 percent of all strategies. Most colleges (13 of 
15) adopted at least one instructional strategy; 11 adopted at least one support strategy; and a 
majority (9 colleges) adopted at least one of each of these two kinds. Institutionwide policy 


'For example, at Sinclair Community College, the Early Support Program provided comprehensive case 
management, including assistance with college applications and financial aid, to students in local high schools 
serving large proportions of low-income students. 
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The Developmental Education Initiative 
Figure 2.1 

Distribution of Focal Strategies Implemented at the 15 DEI Colleges, 

by Strategy Type 


High school strategies 
] Curricular/instructional strategies 


Policy strategies 
J Student support strategies 


Other instructional 
reforms 
8 % 


High school 
strategies 
4% 



Pretesting and 
placement test 
preparation 

8 % 
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changes and high school interventions were less frequently cited as focal strategies. 2 

New Versus Scaled-Up Interventions 

Table 2.2 also indicates whether each focal strategy was new or scaled up from AtD (or 
even earlier) beginnings. 

The strategies were virtually evenly divided between those that were new (that is, de- 
veloped specifically for the colleges’ participation in the DEI) and those that were expanded 
from preexisting interventions. Four of the colleges chose all new strategies; five chose to scale 
up all their strategies; and the remaining six colleges decided to mix new and scaled-up strate- 
gies. While support strategies (tutoring, advising, and placement test preparation) were more 
frequently scaled up, instructional reforms were more frequently new. 

Strategy Objective 

The rightmost column of Table 2.2 includes what is, for the purposes of this report, the 
most important typology: the strategy’s objective. The categorization of strategies by objective 
guides the analyses of scaling up and of outcomes associated with the strategies that appear in 
Chapters 3 and 5, respectively. This typology is especially useful because it is consistent with 
the classificatory scheme used in a recent literature review of what is known about effective 
practices in developmental education. 3 Sorting the DEI focal strategies in this manner enables 
this report both to be informed by and to further inform that body of literature. 

Four principal objectives guide the focal strategies: 

• Acceleration strategies enable students to move more quickly through de- 
velopmental education and into college-level work by shortening the timing 
or content of their developmental coursework; they include computerization, 
individualization, modularization, or compression of courses, as well as pair- 
ings of college- level and developmental courses. 

• Avoidance strategies enable students to avoid unnecessary courses or to 
place out of developmental education entirely; they generally take the form 
of placement test preparation strategies. 


2 As discussed in Chapter 4, some colleges also adopted policy changes that they did not count among their 
focal strategies. The colleges were advised to include as focal strategies ones whose scaling up over time could be 
measured. Policy changes, in contrast, tend to affect all students (or all students subject to the policies) at once. 

3 See Zachry Rutschow and Schneider (2011). 


21 



• Support strategies impart general skills, strengthen specific academic skills, 
or provide academic and/or personal counseling. Some examples of support 
strategies include study skills courses, tutoring/supplemental instruction, and 
case management or advising interventions. 

• Approaches aimed at instructional relevance alter curriculum and/or in- 
structional modalities to make courses more engaging to students; they in- 
clude contextualized, active, or cooperative learning. 

Table 2.3 further defines the general kinds of instructional and support approaches that the DEI 
colleges put in place to advance each of these objectives. The table also lists the specific 
strategies that colleges implemented under these approaches. 4 

Figure 2.2 shows the distribution of focal strategies, by objective, across the 15 colleg- 
es. Of the 40 strategies with an objective that was readily identifiable, 14 strategies fell under 
the acceleration objective, and 19 fell under the objective of support. Four strategies sought to 
help students avoid unnecessary developmental coursework, and three entailed implementing 
new approaches to make instruction more engaging and relevant. 

The objectives of the five focal strategies involving policy changes could not be classi- 
fied using this schema. 5 The two focal strategies involving high school students were also not 
amenable to ready categorization, because such approaches typically pursue multiple goals. For 
example, strategies directed toward low-income high school students may be intended to help 


4 The relationship between strategy type and objective is not straightforward. All strategies that are directed 
toward college students and whose type is classified as “support” also have support as their objective. Instruc- 
tional strategies, in contrast, can serve different objectives. Thus, modularized classes aim at accelerating 
students through the developmental sequence; contextualized curricula seek to make instruction more relevant 
to students; and learning communities that pair a developmental class with a college success class strive to 
offer students the supports they need for college success. 

In classifying strategies according to their objective, it was critical to look at the content of the strategy, 
not just its label. For example, a learning community that was established by linking two courses was catego- 
rized as a support strategy if one of the paired courses was a study skills or “student success” course; it was 
seen as an acceleration strategy if it entailed placement in a college-level course with extra time allocated for 
shoring up basic skills; and it was considered to be a contextualization strategy if a developmental course was 
paired with a college-level course whose subject matter provided the basis for skills acquisition and for 
classroom assignments. That said, researchers at times had to decide between two plausible options. Thus, 
linked courses at Zane State College were deemed to be contextualization strategies, although they also served 
to accelerate progress (because the college-level course was completed in the same term as the developmental 
course, rather than back-to-back). 

5 A policy to bar late registration, for instance, seeks to ensure that students who are already behind aca- 
demically do not fall further behind by enrolling in classes that are already in session. 
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The Developmental Education Initiative 
Table 2.3 


Objectives of the DEI Focal Strategies, General Approaches, 
and Specific Interventions to Meet Each Objective 


Objective and Approach 

Definition 

Colleges' Focal Strategies 

Acceleration 

Approaches that enable students to move 
more quickly through developmental courses 
and into college-level work 


Course computerization and 

individualization/modularization/ 

compression 

Typically, heavily computerized courses 
with instructors available for assistance as 
needed, and students proceed at their own 
pace. A traditional semester-long course may 
be divided into shorter discrete units, or 
modules, that are designed to improve a 
particular skill, with students taking only the 
modules they need to strengthen areas of 
weakness. Alternatively, the contents of two 
developmental courses may be integrated into 
a single, intensive one-semester course. 

Coastal Bend - Fast Track 

Danville - modularized math 

Eastern Gateway - math redesign 

Eastern Gateway — English redesign 

El Paso - Math Emporium 

Flousatonic - open-entry/open-exit English 

Flousatonic - open-entry/open-exit pre-algebra 

Housatonic — open-entry/open-exit algebra 

Houston - math bridge course 

Sinclair - modularized math 

Zane - compressed math 

College-level and developmental 
course-pairing 

Placement of developmental students in 
college-level classes with additional class 
time focused on academic skill-building 

Coastal Bend — linked English 
Patrick Henry - ALP 
Sinclair - ALP 

Avoidance 

Approaches that enable students to avoid 
unnecessary developmental coursework, 
helping them place into a higher level or out 
of developmental education altogether 



(continued) 



Table 2.3 (continued) 


Objective and Approach 

Definition 

Colleges' Focal Strategies 

Placement test preparation 

Short, intensive courses to enable students 
taking (or retaking) the placement test to 
score high enough to place out of 
developmental classes 

Danville — Compass test preparation 
El Paso - PREP 

Guilford Tech — placement test review 
North Central State — math boot camp 

Support 

Approaches that impart general skills, 
strengthen specific academic skills, or provide 
academic and/or personal counseling 


Study skills courses 

Stand-alone or paired with other courses, 
courses to help students develop study skills 
and "habits of mind’’ (for example, time 
management) 

Cuyahoga - paired math and student support 
Houston — GUST (stand-alone) 

Houston — GUST paired with developmental 
math or English 

Norwalk - paired writing and student success 
Valencia - paired developmental education and 
student success 

Tutoring/supplemental instruction 

Academic assistance to aid course-passing. In 
supplemental instruction, assistance is 
provided by students who serve as in-class 
role models. 

Cuyahoga — supplemental instruction 
North Central State — writing tutoring 
North Central State - math tutoring 
Valencia - supplemental instruction 

Case management/mentoring/ 
advising 

Provided by professional staff or volunteers, 
assistance to students in course selection, 
progress monitoring, personal counseling, or 
other services to help students adjust to 
college life. 

Coastal Bend — case management 
Cuyahoga — mentoring 
El Paso - case management 
Guilford Tech - SOAR 
Guilford Tech - advocacy 
Patrick Henry - advising 
Sinclair - Early Support Program 3 
South Texas — face-to-face case management 
South Texas — case management by phone 
or e-mail 

Valencia - high school bridge 3 
Zane - advising 
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Table 2.3 (continued) 


Objective and Approach 

Definition 

Colleges' Focal Strategies 

Instructional relevance 

Approaches that alter curriculum and/or 
instructional modalities to make courses more 
engaging to students 


C ontextualization 

Curriculum that integrates academic skills 
with subject matter covered in college-level 
academic courses or technical courses. 

South Texas — contextualized reading and 
English 

Zane — linked developmental education and 
college-level courses 

Active/cooperative learning 

Instruction emphasizing in-class group work 
and interactive learning 

Patrick Henry - cooperative learning 


NOTES: ALP = Accelerated Learning Program. 

PREP = Pretesting/Retesting Education Program. 

GUST = Guided Studies. 

SOAR = Student Orientation, Advising, and Registration. 

“While these strategies are classified as having a "Support" objective, they in fact had multiple goals. 



The Developmental Education Initiative 
Figure 2.2 

Distribution of Focal Strategies Implemented at the 15 DEI Colleges, 

by Strategy Objective 
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them secure financial aid, to provide counseling, and to strengthen their academic skills so as to 
avoid the need for developmental classes. The analyses in subsequent chapters that deal with the 
focal strategies exclude these seven strategies whose objective cannot be readily classified. For 
some analyses presented in these chapters, additional focal strategies are excluded because other 
data required for the analyses are missing. Thus, different analyses include slightly different 
numbers of focal strategies. 

Chapter 3 discusses the colleges’ experiences in scaling up their focal strategies. 
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Chapter 3 

Scaling Up the Focal Strategies 


This is the first of two chapters on the implementation of the Developmental Education Initia- 
tive (DEI). The chapter responds to a major charge of the evaluation: to assess the extent of 
scaling up of the focal strategies discussed in Chapter 2 and to understand the conditions that 
have facilitated or impeded scale-up at the 15 DEI colleges. To address these issues, the analysis 
employs both quantitative data and qualitative data. 


Key Findings 

• Across the colleges, the proportion of incoming developmental students 
served by at least one focal strategy more than doubled between fall 2009 
(the initiative’s first semester) and fall 2011 (the last semester for which data 
were available), rising from 18 percent to 41 percent. 

• Colleges generally did not meet the ambitious participation goals that they 
had set for serving students with their individual focal strategies. 

• While colleges were not expected to move from serving “more to most” stu- 
dents within the time frame of the DEI demonstration, by fall 2011, three of 
the colleges were able to serve the large majority of their incoming develop- 
mental students with one of their key strategies; 11 colleges, however, 
reached less than half of their incoming developmental students with a focal 
strategy and thus had a considerable way to go in meeting this mark. 

• There was no difference in the extent of scale-up between new strategies and 
preexisting ones. 

• Resource adequacy, communication, engagement, and a commitment to uni- 
form instructional practice were especially important factors promoting 
large-scale implementation of the focal strategies; a college’s small size, by 
facilitating communication, may also have made scale-up easier. 

• Several instructional strategies were notable because they began at scale, 
serving all students from the beginning and reflecting a departmentwide or 
institutionwide commitment to uniform instructional practice. 

• In contrast, factors that worked against full scale-up included institutional re- 
luctance to impose mandates about how students should leam and instructors 
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teach, students’ own needs and priorities, a desire to scale back when strate- 
gies appeared to be ineffective, and a desire to evaluate the strategies’ appar- 
ent effectiveness before moving forward. 


Progress in Scaling Up 

To determine the extent of scale-up across time, this report uses participation data that the 
colleges submitted to JBL Associates. It gauges scale-up in three ways. First, it examines the 
proportion of students at each college who participated in any of the college’s focal strategies. 
Then, it measures the number of students participating in each strategy at a college against two 
standards: the participation goal that the college set for itself and the number of students 
identified as being in the strategy’s target population. As noted in Chapter 1, in the summer of 
2009, after the Kickoff Institute, colleges were asked to identify their focal strategies and to 
specify the target population and the number of students to be served by each focal strategy 
during each of the three subsequent academic years. These figures established the colleges’ 
participation goals during the first two years of the demonstration. Introduced at the beginning 
of the third year, the “more to most” framework supplied a new way to think about scale-up 
objectives — although colleges were not, in fact, expected to serve most students by the end of 
the third year. The use of the two kinds of standards in the analysis of participation presented 
here allows progress to be measured against both sets of goals. 

Scaling Up at the College Level: Participation in Any Focal Strategy 

Table 3.1 shows, for every college but one, the number and percentage of incoming de- 
velopmental students who were touched by at least one focal strategy in their first semester at 
the college. It presents these statistics for two points in time: fall 2009 (the DEI’s kickoff 
semester) and fall 2011 (the most recent semester for which participation data were submitted). 1 
Policy interventions are not included in the calculations for Table 3.1; the table considers only 
interventions that treat students on an individual basis. 2 

Across the 14 colleges represented in Table 3.1, the proportion of incoming develop- 
mental students served by at least one focal strategy more than doubled over the two years, 
averaging from 18 percent to 41 percent. All but one college served a higher proportion of 


] The DEI itself continued through spring 2012, and some colleges may have continued to scale up through 
the spring term. 

2 As discussed below in this chapter, some schools were unable to track individual student participation in 
some strategies; these strategies are also excluded from Table 3 . 1 . One college did not submit participation data 
for fall 2011; the college’s data for spring 2011 are used instead. One college is excluded from Table 3.1 
because its data did not indicate the developmental status of individual incoming students. 
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The Developmental Education Initiative 
Table 3.1 


Proportion of Incoming Developmental Students Served by 
at Least One Focal Strategy, Fall 2009 and Fall 2011 



Fall 2009 


Fall 2011 


College 

N 

% 

N 

% 

College A 

31 

3 

154 

14 

College C 

921 

18 

1,704 

28 

College D 

653 

25 

1,082 

46 

College E 

156 

5 

317 

10 

College F 

64 

63 

145 

98 

College G 

0 

0 

425 

100 

College H 

35 

7 

43 

11 

College I 

0 

0 

170 

7 

College J 

9 

2 

127 

58 

College K 

230 

5 

600 

17 

College L 

511 

7 

3,724 

42 

College M 

42 

11 

40 

14 

College N 

254 

47 

402 

91 

College O 

1,047 

53 

778 

35 

Average 


18 


41 


SOURCE: MDRC and CCRC calculations from colleges' reports to JBL. 
NOTE: Data for College B are unavailable. 


developmental students over time — an increase that is especially noteworthy, given the growth 
in the colleges’ enrollment in general and their developmental populations in particular. 

In addition, three colleges — Eastern Gateway Community College, Patrick Henry 
Community College, and Zane State College — clearly moved from “more to most,” serving 
almost all of their incoming developmental populations with at least one strategy in fall 2011. 
Perhaps the most prominent feature of these three colleges is their relatively small size: All 
three colleges serve fewer than 4,000 students. Being small did not guarantee broad scale-up. 
Another college of comparable size served less than 20 percent of its students. Still, being small 
may have facilitated communication about the DEI among the three successful colleges’ various 
stakeholders and may have supported strong collaborations among faculty and staff. Further- 
more, the infusion of DEI resources may have made a greater difference for colleges that had 
fewer students to serve. 
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In this same vein, a college’s very large size may present sizable but not insuperable 
barriers to scale-up. The Houston Community College System serves some 60,000 students in 
six colleges on eighteen campuses. Each college is led by its own president, who reports to the 
university district chancellor. In this context, scaling up developmental education reform 
requires securing the buy-in of multiple campus presidents and deans within a governance 
structure that is decentralized and that pennits each campus to operate independently. Com- 
municating effectively with faculty across the 18 campuses and maintaining high levels of 
faculty engagement can be challenging. 

Furthennore, successfully engaging a large number of students who are so widely dis- 
persed is no small feat. As an example, Houston advertised its voluntary, non-credit-bearing 
Math Bridge course to eligible students using robocalls and e-mail blasts. However, the college 
found that while these “light touch” outreach efforts were effective for reaching students, they 
were less effective in convincing students to enroll in the course. Administrators concluded that 
students would have benefited from a “heavy touch” outreach strategy in which advisers 
explained to students why the course could help them to advance through the developmental 
sequence. Securing the heavier touch would have required a great deal of cooperation among 
advisers across the campuses, however, and this outreach strategy was not pursued. 

Despite these obstacles, Houston was able to engage a much larger proportion of in- 
coming developmental students in fall 20 1 1 than it had two years earlier. 

In addition, it is worth noting that the three small colleges that reached high proportions 
of students were not able to scale up all their focal strategies to an equal degree. Rather, at two 
of the three colleges, a single focal strategy was the driving force behind the colleges’ ability to 
serve large proportions of their students. When explaining scale-up, it may well be more useful 
to examine the characteristics of the strategies themselves, including the factors that support 
their expansion, than to focus on the characteristics of the individual colleges, which, in any 
event, are fairly intractable. 

Scaling Up Individual Focal Strategies: Comparing the Number of 
Participants with Stated Goals 

A second approach to measuring scale-up compares the goal set for each of 33 focal 
strategies for the 2010-201 1 academic year with the number of students served by the strategy 
in that year. 3 These goals were generally quite ambitious: For example, 10 colleges set targets 


3 The 2010-2011 academic year was selected because a full year of participation data is available for each 
college. (In contrast, only fall data are available for the 2011-2012 academic year.) Moreover, the 2010-2011 
academic year represents the midpoint of the initiative. Most colleges’ targets for this year were at about two- 
thirds the level of their final 2011-2012 targets. If colleges were unable to hit this intermediate target, it seems 

(continued) 
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that met or exceeded the number of incoming developmental students that they had enrolled in 
the 2008-2009 academic year. 

Four strategies reached many more than the number of students originally targeted; in- 
deed, one strategy served double that number, and another four times the goal. The majority of 
strategies did not attain their target goals, however: 1 1 of the 3 1 strategies reached less than a 
quarter of targeted students; seven strategies, between 25 percent and 49 percent; 4 strategies, 
between 50 percent and 74 percent; and 5 strategies, between 75 percent and 99 percent. 

Scaling Up Individual Focal Strategies: Comparing the Number of 
Participants with the Size of the Target Population 

The evaluation uses a third quantitative approach to assess scale-up. In this approach, 
the degree of scale for a given strategy during a given semester is defined as the proportion of 
students identified as being in the target population for the strategy who were actually served by 
that strategy during the semester. Each college had the latitude to define the target population 
for each of its focal strategies; thus, the targeted populations vary across colleges and strategies. 4 

This measure of scale-up affords a perspective on the colleges’ progress in moving from 
more to most targeted students, although, as noted above, the focal strategies were not expected 
to serve most eligible students within the three-year demonstration period. The available data 
permit comparison of the number of students who actually participated in each of 36 focal 
strategies in fall 2011, when the “more to most” framework was introduced, with the number of 
students eligible for that strategy. A statistical definition of “most” is not included in the frame- 
work; if “most” means a simple majority of students, one-quarter of the focal strategies (9 of the 
36) are found to have engaged 5 1 percent or more of eligible students. If the standard for serving 
“most” students is raised to 75 percent or higher, then 5 strategies met that standard. 


unlikely that they would be able to achieve their 2011-2012 targets. The analysis excludes strategies for which 
participation data are missing, as well as strategies for which service targets were not developed (either because 
these strategies were added later in the planning process or for other reasons). 

4 For example, a strategy designed to support learning in mathematics might target first-term developmen- 
tal math students, students (whether first-term or continuing) who have not yet completed developmental math, 
or students (whether developmental or college-ready) who are enrolled in any math course. Moreover, some 
strategies are targeted at very large populations (for example, students who are enrolled in any math course at a 
large community college), while others are targeted at very small populations (such as first-semester students 
who have at least two types of developmental needs and attend a small college). 
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Figure 3.1 shows the percentage of targeted students served by each strategy in fall 
2009 and fall 201 1 , 5 The figure illustrates three patterns of scale-up: 

• For 28 strategies — the large majority — the proportion of the population 
served was initially less than 30 percent; over time, it remained level or in- 
creased slightly, so that the strategy still engaged less than 35 percent of the 
target population by fall 2011. 

• With respect to 8 strategies, the proportion of the population served was ini- 
tially less than 50 percent, but there was a more substantial increase over time. 

• A single strategy reached a very large proportion of the target population in fall 
2009 and continued to serve a very large proportion two years later. 

Factors Associated with Scale-Up: Quantitative Findings 

Quantitative methods were first used to explore whether the characteristics of each intervention 
are related to its expansion over time. Figure 3.2 shows the extent of scaling-up of strategies that 
were new with the DEI and of strategies that had previously been in place at the colleges. For 
both groups of strategies, the figure presents the average percentage of the population served, as 
well as the percentage range, for fall 2009 and fall 20 1 1 , 6 Surprisingly, whether a strategy was 
new or scaled-up does not seem to be associated with the proportion of the population served at 
the demonstration’s outset in fall 2009. Furthermore, new strategies seem to have been just as 
successful as (or more successful than) preexisting ones in expanding to serve a larger propor- 
tion of the population by fall 2011. 

In temis of objective and approach, strategies that fall under the rubric of Instructional 
Relevance were especially likely to reach large proportions of targeted students, as shown in 
Figure 3.3 and discussed below. While mentoring and advising strategies reached substantial 
percentages of participants by fall 2011, qualitative data suggest that some of these strategies 
could be characterized as fairly “light-touch.” 


5 Figure 3.1 includes only strategies that were tracked on an individual basis and for which participation 
information was consistently submitted (N = 37). Appendix A provides further detail regarding how target 
students’ data were collected from each college. 

6 The average percentage is calculated as an unweighted average of all strategies within the category. The 
range represents the minimum and maximum percentages served by all strategies within the category. 
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Factors Associated with Scale-Up: The Qualitative Point of View 

The remainder of this chapter uses qualitative data from interviews, focus groups, and site 
reports to examine the factors that have advanced scale-up of the DEI focal strategies. A 
number of strategies are called out for special attention because they were able to reach very 
high proportions of their target populations. It is worthwhile asking what made the exceptional 
degree of scale-up of these strategies possible. 

Four elements — resources, communication, engagement, and a commitment to uni- 
fonn instructional practice — emerge as especially important factors promoting scale-up in the 
DEI and are discussed below in turn. 7 Factors constraining scale-up are also considered. These 
include the absence of sufficient resources, communication, and engagement. But, aside from 
these “negative” explanations, other considerations also limited the degree to which focal 
strategies were expanded. The qualitative analysis makes it clear that competing values some- 
times came into play in influencing decision-making about scaling up — or scaling back. 

Resource Adequacy 

Staff, space, and technology all fall under the “resource” rubric. Resources affect not 
only the number of students who can be served but also the depth and intensity of those ser- 
vices. Resources need to be commensurate with stated goals. Otherwise, the scope of these 
goals is likely to contract to align with the resources that are available. 

Staff 

In November 2011, Zane State College served 1 00 percent of its students eligible for in- 
tensive advising. The college had implemented intensive advising for developmental math 
students under the Achieving the Dream (AtD) initiative. Under the DEI, the college scaled up 
this strategy to cover all developmental students. What made this possible was the deployment 
of two case managers (one of whom was funded by the grant) whose work was especially 
focused on high-risk students and on specific tasks. An important part of the case managers’ 
work was to identify students who had preregistered for courses for which they had unmet 
developmental prerequisites and to register them instead in the courses that they were first 
required to take. Another charge was to identify students who were qualified to enroll in the 
learning communities and compressed math courses that constituted the college’s acceleration- 
oriented focal strategies. The case managers contacted and worked with students in an early 


7 Versions of the first three of these elements are among those identified in a conceptual model known as 
SCALERS, which seeks to identify the elements that are essential to expanding any initiative. The SCALERS 
model was developed by Bloom and Chatterji (2009) and was adapted by MDC to make it more relevant to 
community colleges and to the DEI. 
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intervention program for students identified as having attendance or performance problems. 
Finally, the case managers paid special attention to students whose profiles on the Noel-Levitz 
Retention Management System’s College Student Inventory (CSI) identified them as at high 
risk of dropping out. 

During the last year of DEI implementation, Zane State College’s ability to provide in- 
tensive advising was challenged by turnover in the advising staff and by a sizable influx of new 
students who required developmental coursework and who were identified as being at high risk. 
The college responded by hiring three paraprofessional advisers from among the developmental 
education faculty to continue to provide intensive and personalized counseling to students. In 
this way, the college was able to reach all eligible students. 8 

South Texas College’s record in scaling up its case management component stands in 
contrast both with Zane’s scale-up of intensive advising and with its own more successful 
implementation of contextualized learning, discussed below. At South Texas, case managers’ 
initial workloads simply proved excessive. At first, the college planned to have four case 
managers contact new, first-time-in-college developmental reading and English students four 
times during the first semester and three times during the second semester. The case managers 
who were hired were judged to be highly skilled and efficient, but their caseloads were too large 
— case managers at the main campus had caseloads of 600 students each — to allow for the 
face-to-face interactions with students that were deemed most desirable; instead, a contact was 
defined as any instance in which the case manager has initiated interaction with the student 
(whether in person or by phone or e-mail), and gets a response. Faced with financial constraints 
that precluded the hiring of additional case managers, administrators decided that resources 
were spread too thin to be effective. They opted to scale back rather than up by limiting case 
management to first-time-in-college students who had attended an orientation session. 

Cuyahoga Community College’s experience with respect to mentoring is somewhat 
akin to that of South Texas College. Cuyahoga initially planned to recruit faculty members, staff 
members, and administrators as volunteer mentors and to assign them to students on a one-to- 
one basis. The number of students requesting mentors far exceeded the number of mentors, 
however. In response, the college allowed students to serve as mentors, had small groups of 
mentors work with larger groups of students, and assigned mentors to entire classes. While 
these stratagems for stretching mentoring resources enabled more students to gain exposure to 
an individual who could help them resolve problems (or direct them to someone else who could 

8 While Zane’s experience is described here to illustrate the importance of resource adequacy, another fac- 
tor that undoubtedly played into successful scaling of the strategy was the close working relationship between 
the director of the DEI and the director of the college’s Student Success Center, to whom the advisory staff 
reported. The two were long-term colleagues, communicated frequently, and held similar views about the 
strengths and needs of developmental students. 
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help them), constraints on these resources changed the nature of mentoring, making it a “lighter- 
touch” component than had originally been intended. Internal college data suggested that the 
intervention was nonetheless having a positive effect on student retention, and the college is 
continuing to experiment with various ways of using peer mentors. 

Technology and Space 

Several of the DEI colleges chose to scale up technology-based instructional strategies. 
In one case, front-end investment in technology was used to expand the reach of what had been 
a classroom-based intervention. Guilford Technical Community College created COMPASS 
Review 2.0, an Internet-based version of a course to prepare students for taking or retaking the 
COMPASS placement test. The online review featured, in high definition, 33 math, 15 reading, 
and 15 writing instructional videos. With the support of various college faculty and staff — the 
developmental education faculty, counselors, assessment center personnel, and institutional 
research and eLearning staff, among others — the online tool was marketed aggressively to 
students to increase participation in the review and to attempt to get students to take the place- 
ment test more seriously. To reach even more students, the college packaged COMPASS 
Review 2.0, placed it on an online educational platform, and shared it free of charge with 
community colleges in North Carolina. By using technology in this way, Guilford Tech was 
able to make the test preparation strategy almost infinitely scalable both inside and outside the 
college, at marginal cost. 

In other cases, scaling up technology-based strategies required that colleges make avail- 
able to faculty and students the physical space and equipment necessary to secure access to that 
technology. To further its goal of integrating technology into all its developmental education 
courses, Danville Community College not only built a new computer lab for math instruction 
but also used Title III grant money to purchase two mobile labs to support the use of technology 
in the classroom. In addition, to foster faculty and student engagement in technology-based 
instruction, the college revised its course schedule to increase the amount of available lab time. 
Norwalk Community College used the DEI grant as an opportunity to designate a computer lab 
for a new one-credit Informational/Technology literacy course that was linked to students’ 
learning communities and in which students were introduced to electronic portfolios. 

To expand its computerized, self-paced instructional model known as Math Emporium 
at four of its five campuses, El Paso Community College reconfigured, remodeled, or built 
additions onto existing structures. At the fifth and largest campus, the college chose to build a 
pennanent, freestanding facility to house Math Emporium classrooms. Unexpectedly, progress 
was stalled by the introduction of a new city ordinance that caused major construction delays 
and forced the college to incur a significant financial burden in order to meet the city’s new 
requirement. Committed to the Emporium model for math and other disciplines, the college 
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continued to move forward with plans to build the facility, despite the added cost. Until the new 
lab could be completed, the college continued to offer Emporium-style courses using computer 
carts in temporary classrooms. 

Some DEI colleges that chose to scale up technology-based focal strategies did not 
mandate student use of the technology, as discussed below. Thus, technology was a factor that 
enabled scale-up, but, absent other conditions, its availability did not ensure that large numbers 
of students would pursue interventions involving computerized instruction. 

Communication and Engagement 

As in the 201 1 report, 9 Communication and Engagement are discussed together because 
the concepts are highly interconnected. Communication efforts described here focused on 
informing campus constituents about the DEI in the hopes of engaging and encouraging them to 
support or adopt the focal strategies. 

Strong and positive communication enables successful scale-up, and college presidents 
played a critical role in communicating to various constituencies the importance of the DEI in 
general and of the college’s focal strategies in particular. As the initiative progressed, these 
leaders continued to play an important role in communicating to the college’s faculty consistent 
messages about the institution’s ongoing commitment to the DEI. At Danville Community 
College, during interviews with prospective faculty members, the president promoted the 
college’s participation in the national initiative — partly to demonstrate the college’s commit- 
ment to its developmental learners and partly to help prospective faculty understand the nature 
and challenge of the work ahead. In his words, “We tell those prospective faculty: ‘This is what 
we do, this is who we are. We want you to be a part of this. If you don’t think you can be a part 
of this heavy lifting, [this may not be the right place for you].”’ This message is also reiterated 
during the college’s faculty/staff orientation. The president also recalled an instance when he 
had to correct misinformation about the initiative after an administrator told several colleagues 
that the college would not continue with the DEI. Reflecting on his role as leader, he comment- 
ed, “I had to publicly state in several arenas that we were not backing off one inch. They needed 
to hear it from me.” 

Smooth scale-up of the focal strategies also hinges on the colleges’ ability to maintain 
engagement among stakeholders. When there is a base of support among the faculty and 
administrators who are charged with scaling up the strategies, the process is easier. According 
to one DEI coordinator, “[Faculty] have to be involved, and it is better if there is lots of faculty 


9 Quint et al. (2011). 
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buy-in.” Similarly, if a strategy requires major changes, such as blocking enrollment or ending 
the policy of late registration, a high level of administrative support is required. 

With this goal in mind, colleges used various methods to engage faculty. Professional 
development and access to technical assistance — topics considered in Chapter 4 — were 
among these methods. Other techniques included involving faculty and staff on teams and on 
committees designed to support scale-up, raise awareness, and advance collaboration. El Paso 
Community College paired DEI with its collegewide Start Right Initiative, which was led by 
four work groups composed of faculty, staff, and student representatives, all engaged in efforts 
to promote student success. El Paso also engaged faculty through a Student Success Core Team, 
whose efforts to scale up and institutionalize the focal strategies received the attention of key 
leaders and decision-makers at the college. Cuyahoga Community College created teams of 
faculty and staff who collaborated on drafting the job description for the supplemental instruc- 
tion leader positions that the college sought to fill; team members also participated in the 
interview process. 

The scale-up process also benefited in instances where faculty (and other key actors) 
took ownership of a given strategy. At El Paso Community College, a math professor involved 
in the scale-up of the Math Emporium strategy created a set of online videos to help other math 
professors practice teaching in an Emporium-style classroom. Similarly, at Houston Communi- 
ty College, a math professor developed online videos to help students in the Math Bridge course 
prepare for their final exams. And, at Housatonic Community College, the director of academic 
support, the lead developmental math professor, and the college’s testing coordinator worked 
together to develop instructional materials for the college’s placement test preparation strategy. 

Proactive efforts to engage faculty could be complicated by a strong faculty union, as 
was the case at Norwalk Community College. At that institution, incentives persuaded a cadre 
of dedicated adjuncts and full-time instructors to lead a notable expansion of the college’s 
learning community strategy. 

Some colleges that were not as inclusive in engaging faculty early on — such as in the 
grant-writing process or in the development of the focal strategies — continued to struggle with 
securing faculty buy-in and maintaining it. According to one DEI coordinator, when time- 
limited grants are written without faculty input, scaling up the strategies can be very challenging 
because time is needed to build support for the work. Absent that strong support, a lot of time 
can be spent trying to persuade faculty of the strategy’s value. In one case, a DEI college scaled 
up an instructional strategy that paired a college math course with a student support course. 
According to the DEI coordinator, the decision to scale up the strategy was taken “on high,” and 
success depended on how the various deans communicated with faculty about the strategy. In 
the beginning, a lack of communication with the administration and among faculty at the 
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college’s several campuses led to the course’s being offered in inconsistent formats across the 
campuses. The strategy also met resistance among some faculty members who had not bought 
into it. As a result, the college chose to slow scale-up until the faculty and administration could 
agree on the appropriate student population to be targeted for the strategy. 

The DEI colleges’ outreach also extended to students and external stakeholders. As 
mentioned in the 2011 report, several colleges launched significant marketing campaigns to 
promote and engage current and prospective students. At Coastal Bend, DEI was a key topic at 
convocation. In this same vein, Patrick Henry Community College created an award-winning 
marketing campaign that included developing a logo to represent the initiative, selecting a 
theme song and campaign name, and organizing two public events that featured nationally 
known figures as guest speakers and that received coverage by local and regional print and 
broadcast media. 

Efforts to engage stakeholders were not limited to current college students. Danville 
Community College used persuasive marketing to high school seniors — and their parents — to 
get eligible students to take up its test preparation strategy. The messaging emphasized the 
benefits of test preparation in terms of both saved dollars and saved time. Parents were more 
responsive than youth, and they were able to encourage their children to participate. In this case, 
parents were influential actors in the scale-up of the test preparation strategy. 

There were also cases where college personnel felt that student marketing fell short and 
that the implementation of the strategies suffered as a result. As noted above, Houston Commu- 
nity College officials believed that they had had only limited success using robocalls and e-mail 
notification to market the college’s Math Bridge strategy to students. At Sinclair Community 
College — which scaled up modularized math — advising and online registration issues 
surfaced when students registered for the modularized classes unaware that the course was self- 
paced. As a result, the college had to revisit how the course was described to students. Among 
other things, the college’s journalism department put together a glossy publication in which the 
DEI focal strategies were featured. Journalism students interviewed students participating in the 
college’s DEI interventions, who talked about the benefits of enrolling in the new courses. 
Thus, a class project for one course became an important recruitment method for other courses. 

Commitment to Uniform Instructional Practice 

Because teaching and learning are the core activities of community colleges, it is es- 
pecially relevant to examine the factors contributing to the wholesale expansion of four 
instruction-related focal strategies. The redesign of developmental math and English at 
Eastern Gateway Community College and the contextualization of instruction at South Texas 
College each reached over 85 percent of their target students in fall 2011, and collaborative 
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learning at Patrick Henry Community College engaged over 70 percent of its target group in 
the same semester. 

Two aspects of the scale-up of these strategies are especially notable. First, in the case 
of developmental math and English redesign at Eastern Gateway and of contextualization at 
South Texas, the interventions were planned to reach all students from the outset. Rather than 
scaling up, these initiatives began at scale. The students were already “on board” to take the 
new courses; no efforts to recruit or engage them were necessary. Faculty members, too, were 
on hand to teach the courses; only how the courses were taught changed, so that implementing 
the strategies required little in the way of additional staff resources. 10 

Second, at all institutions, the departments involved — and, in the case of Patrick Hen- 
ry, the institution as a whole — were all committed to having all faculty members teach the 
courses in a certain way and to having all students taking the courses be exposed to those modes 
of learning. How, and how easily, this commitment was attained varied considerably, however. 

The scaling-up at Eastern Gateway of an individualized, computerized approach to 
math instruction and at South Texas College of contextualized instruction took place through a 
process that might be termed “gentle fiat”: Administrators — sometimes in conjunction with 
faculty, sometimes not — make a decision that all classes will be taught in a certain way, and 
faculty members are informed as to how they are expected to teach. While instructors may have 
input into the new approaches and may receive professional development related to implement- 
ing them — this is what makes the fiat “gentle” — they are not free to deviate from them. 

At Eastern Gateway, administrators were dissatisfied with the low success rates of de- 
velopmental courses and were determined to do something different. Early in the demonstra- 
tion, a top official commented that while faculty members were concerned about student 
attendance and performance, they tended to see the problem as one of student motivation and 
preparation rather than their own instruction; in her view, they needed to alter their teaching 
practices, which focused on lecturing, to better meet students’ needs. The first year of the grant 
was a preparatory one. A team of outside consultants advised the developmental math instruc- 
tors to replace lectures with a math “lab” approach, in which students work independently on 
the computer and teaching staff are on hand to provide additional explanations when students 
cannot figure out the material on their own. Eastern Gateway faculty members were given a 
choice about which computerized product to use, and they elected to make MyMathLab — a 
computerized instructional product developed by Pearson Education, Inc. — the centerpiece of 


l0 In contrast, scaling up support strategies like case management may require new resources rather than 
the redeployment of resources that are already available. 
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the new instruction, since the program had already been used in the practice component of their 
lecture courses; instructors also received training on the use of MyMathLab. 

Initially, the change encountered resistance on the part of some faculty members, who 
were unwilling to give up lecturing as a teaching modality (and who, according to some of their 
colleagues, were not entirely computer-literate themselves). A compromise was reached: 
Faculty were permitted to deliver a 10-minute “mini -lecture” at the start of class. Nonetheless, 
the launch of the reform in fall 2010 was a rocky one, and many implementation problems were 
reported; the second year of implementation went far more smoothly. 

Despite all these issues, and in the face of turnover in project leaders, Eastern Gate- 
way was consistently able to serve a majority of the students targeted for the developmental 
math focal strategy. The course redesign’s scale-up was also supported by the college’s 
decision to institute a policy mandate prohibiting students from enrolling in college-level 
courses until all developmental education requirements had been completed. The mandate 
guaranteed that all developmental learners would have to enroll in developmental education 
courses first and that, once enrolled, students would automatically be served by the new 
computerized instructional format. 

Contextualization of developmental reading and English at South Texas College had a 
much smoother reception. For one thing, the approach, while mandated by the administration, 
arose from the faculty. Faculty members in the two departments had been appointed to a 
committee that was charged with identifying workable strategies for the DEI grant application, 
and, after conducting research on best practices in the fields of developmental reading and 
writing, they arrived at the idea of curriculum contextualization. For another thing, the concept 
of incorporating topics related to history (a subject that students were required to take) and to 
sociology (a subject that many students took anyway) into reading and writing assignments 
struck a resonant chord with faculty members and did not represent a marked deviation from 
past practice. The textbook used in developmental reading classes, for example, included 
selections related to these subjects as well as to anthropology, psychology, and science; the 
developmental English department, in particular, had a history of making changes to the 
curriculum, so that its members took another set of changes in stride. 

A small faculty committee began planning for the new curricula during the summer of 
2009, when many faculty members were away. There was some irritation on the part of instruc- 
tors when they returned in the fall to find a fait accompli. They offered no strong resistance to 
implementing the new contextualized units, however, and, over time, both reading and writing 
instructors came to feel that the contextualized assignments were helping students to succeed in 
their college-level coursework. 
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The plan was for one of the five major assigmnents in each writing course (four major 
assignments in the course for the lowest-level students) to involve a contextualized topic in 
Year 1 and to add a second such assigmnent in Year 2 and a third in Year 3. Thus, during the 
first year, students in the lowest-level developmental English course wrote an essay on the 
benefits of a college education; midlevel students wrote about gender roles in society; and the 
highest-level students wrote about multicultural education. Upon reassessment, faculty mem- 
bers decided to limit the number of contextualized assigmnents to just two in the DEI’s third 
and final year. For one thing, in a data-oriented institution, data from the third year would be 
collected too late to inform future decision-making. Perhaps more important, instructors 
discovered that the contextualized writing prompts required more time and effort of students, 
who had to weave together information from a variety of sources as they wrote their essays. 
Adding a third contextualized assignment while keeping students’ workloads manageable 
would have required that instructors reduce further either the length or the total number of 
essays that students were to write; instructors were willing to do neither. Instead, faculty cut one 
major essay from each course’s requirements, substituting shorter writing tasks. In a sense, then, 
resource constraints, in the form of available time, affected the depth of contextualization, 
although not the number of students affected by the strategy. 

If the commitment to computerized and individualized math instruction at Eastern 
Gateway came from the top down, the decision to adopt the same approach for developmental 
English at the college came from the bottom up. The DEI project director urged members of the 
English department to develop their own reform strategy, telling them that this might be the last 
and best opportunity to do so that they might have for some time — and especially before the 
State of Ohio, which was in the process of reexamining all aspects of higher education, might 
lay down some mandates that Eastern Gateway faculty might dislike. The faculty elected to 
teach writing using a computerized approach, using another Pearson product, MyWritingLab. 

Finally, the impressive expansion of collaborative learning at Patrick Henry Communi- 
ty College (from 29 percent of the target population reached in fall 2009 to 7 1 percent reached 
in fall 2011) reflected an institutionwide commitment to this mode of instruction, with which it 
had begun to experiment in 2003 as an early Achieving the Dream college. By the time the 
college was awarded a DEI grant, more than half of all faculty members and 1 00 percent of 
developmental education faculty members had completed extensive training and had adopted 
cooperative learning strategies for use in their classrooms. The principle of using active cooper- 
ative learning to enhance student engagement inside the classroom had been embraced through- 
out the college. In addition, the college had emerged as a national leader in active cooperative 
learning for community colleges and was in the process of founding its own institute to offer 
public schools and postsecondary institutions leadership and training in this area. In essence, 
collaborative learning put Patrick Henry on the map. 
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Student Choice in How to Learn 

While approaches that entail a uniform approach to teaching and learning may promote 
maximum scale-up, they do not accommodate student choices or preferences about what or how 
they leam. In contrast to Eastern Gateway, where all developmental classes are taught in a 
computerized and individualized format, other colleges — including those with well-equipped 
math computer labs — did not push “one size fits all” policies and practices. 

Sinclair Community College took a measured approach to expanding its modularized 
math courses. Students facing developmental math requirements either could select a lab setting 
in which they were expected to work independently on computers, with tutors available to 
provide help when necessary, or could enroll in a traditional math class taught largely through 
lectures. Staff members strongly supported giving students such choice: Instructors maintained 
that independent learning worked for some students — particularly those who were computer- 
savvy and motivated to do more in a given time period — whereas the traditional lecture 
approach worked best for others, including those who were not easily engaged by self-paced 
learning. While instructors and advisers encouraged students to consider the modularized 
approach, they also firmly believed that students should have a voice in selecting the instruc- 
tional modality best suited to their particular learning style. 

Similarly, Norwalk Community College strongly encouraged but did not mandate first- 
time students who placed into developmental English to enroll in learning communities. The 
college conducted small-group orientation sessions at which students heard about the value of 
learning communities and could, if they chose, register in one. 

Other Student Needs 

While many college students take into account when classes meet in choosing their 
courses, time is an especially precious commodity for community college students, who often 
must balance studies with jobs and family commitments. Back-to-back class periods (an 
integral part of some acceleration strategies that involve students enrolling in a college-level 
course immediately preceded or followed by a class focused on skills-building) or course 
schedules that left large gaps during the day did not work for many students. Scheduling posed 
special problems in smaller colleges and colleges in more rural areas (often the same institu- 
tions), where many students had to travel long distances to attend school — sometimes in old, 
unreliable vehicles, according to one DEI coordinator. Especially when a small college’s 
instructional strategy was intentionally targeted to a specific population — such as students 
whose placement test scores placed them in the upper ranks of developmental education 
enrollees — the relatively small number of such students meant that only a few sections of the 
course could be offered, further complicating scheduling considerations and limiting scale-up. 
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Furthermore, economic priorities could trump educational aspirations. When, for exam- 
ple, there was a surge in demand for workers in the natural gas industry, overall enrollment at 
Coastal Bend College dropped, precluding further scaling of a focal strategy aimed at helping 
upper-level developmental students advance quickly and successfully to college-level English. 
The experience of Coastal Bend and other institutions suggests that the more constrained the 
target group that an intervention seeks to serve, the more that scaling up may be affected by 
forces that are largely beyond the college’s control. 

Rethinking Ineffective Strategies 

While Guilford Technical Community College, as noted above, experienced considera- 
ble success in expanding its technology-based test preparation strategy, its record with respect to 
another focal strategy, intensive advocacy, was one of trial and error. In a continuing effort to 
make its advocacy component more effective, the college actually cut back on the scale at 
which the component operated. 

During the first year of the grant, all students who were required to take two or more 
developmental courses were assigned advocates, but many did not respond when the advocates 
attempted to contact them. The next year, administrators decided to direct advocacy to students 
in a study skills class. Embedded in the course curriculum was a series of assigmnents that 
called for students to contact their advocates (for example, an assignment to conduct and write 
up an interview with the advocate). The college discovered, however, that students were short- 
circuiting these requirements by having one student conduct the interview and then share the 
information with others. 

Eventually, administrators concluded that students would contact their advocates only 
when facing a crisis. This led them to designate yet another target population for their advocacy 
efforts: students whose poor academic records (a grade point average of less than 2.0 and 
completion of less than two-thirds of the courses they had attempted) placed them at risk of 
losing financial aid. Within this group, they opted to focus efforts on students who were 
required to take two or more developmental courses and who, with more support, could raise 
their academic standing enough to avoid losing financial assistance. By the end of the third year, 
officials believed that the advocacy component, while serving fewer students than originally 
planned, would be more successful than in the past. 

The Limits of Scaling and the Need for Evaluation 

Added to these considerations is the fact that, as early Achieving the Dream colleges, 
the DEI institutions took data seriously. Some colleges intentionally opted for slower scale-up 
so that they could build up their data collection infrastructure, get preliminary feedback, and 
adjust their strategies as necessary. Colleges with weaker institutional research systems were 
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particularly concerned about setting up the necessary processes for data collection before 
forging ahead with the interventions. 

The call for additional research also reflected the recognition that scaled-up interven- 
tions might prove less effective than the pilot programs that preceded the scaling efforts. When 
asked about the most important lessons about scaling up interventions that he had learned 
through DEI participation, one college president commented: 

Everybody talks about the challenges with scaling. But the other challenge is get- 
ting false readings on our initiatives. We see many false positives — you do this 
pilot project, and success rates are so much better, and then it doesn’t happen at 
the larger scale. 

His counterpart at another institution voiced a similar view: 

When you get to scale, people tend to lose the discipline of data. It’s even more 
important to hold them to standards when you get to scale than when you’re go- 
ing to scale. 

Some colleges understood that rapid scale-up might foreclose opportunities for careful 
evaluation. Patrick Henry Community College focused its efforts on making active and cooper- 
ative learning a central feature of its developmental education program. While the first few 
courses to introduce this approach could be easily compared with those that did not, the contrast 
became less clear as active learning techniques spread throughout the institution. Similarly, the 
DEI coordinator at Coastal Bend Community College expressed regret that the case manage- 
ment strategy had gone to full scale at the start of the DEI. She wished that the college had 
instead implemented a round of baseline data collection or phased the program in semester by 
semester so that there would be a basis for comparing outcomes. 

These examples suggest that a number of the colleges did not automatically see engag- 
ing all students in a focal strategy’s target group as the ultimate good. Instead, administrators 
wanted to be sure that strategies that produced promising results when implemented on a small 
scale would produce equally good or even better results when expanded to reach more students. 
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Chapter 4 

Beyond Scaling: The DEI’s Larger Meaning 


So far, this report has centered on the extent of scale-up of the interventions that the Develop- 
mental Education Initiative (DEI) colleges defined as their focal strategies. 1 But an evaluation 
that is centered exclusively on the expansion of the focal strategies is overly narrow, for a 
number of reasons. First, as discussed in Chapter 3, some of the college administrators came to 
believe that serving all the students in their target populations was not feasible — and, in some 
cases, not desirable. Second, focal strategies that involve policy changes are not well suited to 
demonstrating gradual scaling, since they are generally meant to affect all students (or all 
students within a designated subgroup) all at once. Third, the colleges implemented or scaled up 
additional policies and interventions for students and also directed a good deal of professional 
development toward faculty and staff. Finally, for at least one college, the DEI was the impetus 
for a broader change in campus culture. 

In short, the whole of DEI is more than the sum of its measured focal strategies. This 
chapter considers that whole, examining what the DEI has meant for the colleges, beyond the 
expansion of discrete interventions. 


Key Findings 

• The DEI colleges put in place a number of new policies (some counting as 
focal strategies, others not) aimed at helping students accelerate through and 
otherwise succeed in their developmental courses. 

• DEI funding also supported programmatic interventions and refonns beyond 
those included in their focal strategies. 

• The colleges have used their DEI grants to support on-campus professional 
development on a broad range of topics related to developmental education, 
including the use of new instructional modalities, the characteristics and 
needs of low-income students, and how instructors can help students better 
meet those needs. 


'Tables 2.1a and 2.1b in Chapter 2 identify the DEI colleges, and Table 2.2 presents their focal strategies. 
Chapter 3 discusses the extent of scaling up of the focal strategies at the colleges. 
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• College personnel also learned from each other and from their counterparts at 
non-DEI colleges at conferences and meetings whose attendance was made 
possible by DEI funding. 

• At some colleges, the DEI stimulated wider discussions about student suc- 
cess and campus priorities. 


Policy Initiatives and Scaling 

At the outset of the DEI, colleges were advised that the interventions that they designated as 
their focal strategies should be ones whose scale-up over time could readily be measured. Policy 
changes do not fit neatly into this category, since, once put in place, they generally affect all 
students who are subject to them. 

Some colleges elected to list policy changes among their focal strategies, but they did 
not consistently submit data on the numbers of students affected by these policies. At Coastal 
Bend College and Eastern Gateway Community College, students were required to remain 
continuously enrolled in and complete their developmental coursework before moving on to 
college-level classes. North Central State College revised its policies and procedures related to 
assessment and placement. It instituted counseling for students about the importance of taking 
the initial placement tests seriously. It also allowed some students who, on the basis of their test 
scores, would previously have been placed in developmental English classes to enroll instead in 
college-level courses that had additional instructor-led sessions in which the students could 
strengthen their writing skills. Danville Community College banned late registration for 
developmental students, since evidence suggested that students who started their courses late 
had difficulty catching up with their peers. (The college’s newly modularized math classes 
meant that students who were blocked from registering could enroll in a math module within 
just a few weeks, rather than having to wait months for the new semester to begin.) In sum, 
these policy changes, while not well captured by the scaling numbers in Chapter 3, nonetheless 
shaped the environment in which students enrolled in and attended college and may thereby 
have affected student success. 2 


2 lt is also possible that some of these policy changes have had unanticipated negative consequences. For 
example, a blanket requirement that students complete all developmental coursework before taking any 
college-level courses might be expected to lead to discouragement and dropping out or “stopping out.” The 
data to investigate this possibility are not available. 
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Additional Policies and Programs 

While colleges were asked to identify and center their efforts on the focal strategies, their 
programmatic and policy initiatives under the DEI went beyond these strategies. 

Valencia College offers perhaps the most striking example of this. While Valencia 
listed learning communities and supplemental instruction as two of its three focal strategies and 
reported on the extent to which participation in these activities increased during the DEI grant 
period, it did not use DEI funding to support these interventions. Rather, the college had 
instituted these programs under Achieving the Dream (AtD) and had committed to using 
college funds to institutionalize them upon the expiration of that grant. 

Valencia did use part of its DEI funding to strengthen the third focal strategy: the 
Bridges Program to assist low-income, first-generation high school seniors in making the 
transition to college. The Bridges manager noted that being part of the DEI had given his 
program heightened visibility, while another high-level administrator commented, “In any 
institution this large, to fight for resources, attention, time, just to get a request for programming 
(which may be #479 on someone’s list) — those are not small hurdles. I have no question that 
DEI paved the bridge for [clearing] those hurdles.” 

Most of Valencia’s funding, however, was used to support other initiatives that were 
not counted as focal strategies. These included the creation of the REACH program (a special 
intervention for students who were required to take developmental courses in all three areas — 
reading, writing, and math) and, as discussed in the next section, the implementation of a new 
and ambitious professional development system for faculty. 

As the DEI demonstration moved forward, colleges continued to develop new policies 
designed to speed the progress of developmental students. Coastal Bend, for example, reduced 
the number of courses in its developmental math sequence from three to two, so that students 
could move through the required math coursework more quickly, and South Texas College also 
piloted and then implemented a two-course sequence, first in math and then in reading and 
writing as well. Beginning in spring 2013, developmental math at Eastern Gateway, which is 
already heavily computerized and individualized, will also be modularized, so that a student 
need take only those modules required for his or her particular course of study. At Guilford 
Technical Community College, where grant funding underwrote the development of online and 
in-person placement test review courses, the college was able to institute a new policy requiring 
all students who wanted to retake the COMPASS first to enroll in the review course. 

South Texas College adopted another policy change designed to promote acceleration 
through developmental education. While all developmental students had previously been 
permitted to pay for and take the Accuplacer test at any time during the semester, with the 
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possibility of exiting the course before the end of the term with a passing grade, few had taken 
advantage of this opportunity. For the 2011-2012 academic year, all developmental English 
students were required to take a free Accuplacer writing test after they had completed approxi- 
mately 90 percent of the course in which they were enrolled; their grade on the Accuplacer 
counted as part of their course grade. This resulted in a sizable number of students being able to 
bypass some of the courses in their prescribed developmental English sequence. 

The DEI supported programmatic efforts beyond the focal strategies at other colleges as 
well. El Paso Community College put in place an Early Alert system — an intervention that 
entailed training faculty and securing their full commitment as well as having a system for the 
timely collection and reporting of data. At Danville Community College, the DEI grant not only 
supported the development of modularized math courses but also paid for two tutors who 
staffed the lab at all times. Cuyahoga Community College undertook a new initiative to empha- 
size cooperative learning in the classroom. DEI funds were used to equip classrooms with 
movable desks and multiple whiteboards and easels that would facilitate a new emphasis on 
cooperative learning. 

In Connecticut, both Housatonic Community College and Norwalk Community Col- 
lege undertook reforms in their developmental math programs. Housatonic instituted the iMath 
program — a computer-based skills refresher program that students may take if placed into a 
developmental math course. Once students complete the program, they may retake the place- 
ment test. In the three years that the college has been operating the program, 75 percent of 
iMath participants retaking the test have either moved into a higher developmental class or 
placed out of developmental math entirely. The success of iMath has led the college to pilot an 
iEnglish program, with promising early results. 

Norwalk faculty members worked to better align the courses in the college’s develop- 
mental math sequence and took other measures to improve student performance. After analyzing 
the data to determine which students were succeeding and which were having difficulty, they 
decided to move the highest-level developmental math course into the math department and to 
teach that course in sequence with the gatekeeper math course, using the same text and assess- 
ment practices, among other things. Developmental math faculty also agreed to adopt a common 
curriculum across the courses they taught. And because the data indicated that course repeaters 
were also likely to do poorly the second time around, the college identified a faculty member to 
work both with other faculty members and with students on strategies to improve success. 

At El Paso, math faculty members also examined test scores — in this case, scores on 
the gatekeeper math test scores that former Emporium students had taken. This led faculty 
members to realize that some students, because of their course of study, did not have to take the 
gatekeeper math course at all. This recognition has led to regular meetings between math staff 
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and to better advising, and students are now in a better position to take courses that help them 
advance rapidly toward a credential — and avoid courses that make for unnecessary delays. 


Professional Development Opportunities 

The DEI has presented new opportunities for staff professional development. To better imple- 
ment their focal strategies, as well as to obtain more general feedback, the colleges made use of 
technical assistance provided to them under the DEI grant, drawing on the services of consult- 
ants whose efforts were coordinated by the Community College Leadership Program at the 
University of Texas at Austin. Thus, for example, two experts on contextualized learning 
conducted workshops for faculty members at South Texas College to help them implement 
contextualized curricula in developmental reading and English courses. One of these consult- 
ants similarly helped Zane’s faculty members who were involved in learning communities that 
paired developmental and college-level level courses to recognize the importance of integrating 
the curricula of the two classes. Likewise, Norwalk Community College drew on the expertise 
of a consultant to promote faculty members’ use of integrative learning strategies in learning 
communities that linked developmental and student success courses with a newly created 
Infonnation/Technological Literacy course. 

Colleges also used DEI resources to purchase professional development from other 
sources and for other DEI-related purposes. Cuyahoga Community College brought trainers 
from the University of Missouri at Kansas City to Cleveland so that instructors could leam 
about the supplemental instruction program that the university had developed. Zane State 
College invited to the campus a consultant who led what was generally considered a provoca- 
tive and thoughtful workshop on teaching students in poverty. The consultant emphasized the 
importance of personal relationships to such students; because, in her view, low-income 
students have underdeveloped long-term planning skills, she suggested teaching strategies like 
breaking up an assignment into smaller segments, each with its own goal and deadlines. Zane 
also operated a series of weeklong summer workshops on using technology to promote class- 
room engagement; faculty members, who received a stipend for attending, listened to col- 
leagues with expertise in this area and also had the opportunity to plan for using technology in 
their own classes. Housatonic Community College held an on-campus two-day faculty training 
session focused on student engagement, which was so well received and was reported to be so 
beneficial in the classroom that subsequent two- and three-day sessions were held to expand the 
training and enhance the lessons learned. North Central State College procured technical 
assistance to train faculty and staff in developing logic models for use in the planning, imple- 
mentation, communication, and especially evaluation of key initiatives. Additional training at 
the college focused on enhancing faculty and staff capacity so that all the college’s programs 
and initiatives, including those in the larger strategic plan, could be consistently evaluated. 
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At Valencia College, DEI funding has supported a professional development effort 
aimed at the entire developmental education faculty. The initiative involves training faculty on 
the college’s developmental advising model, known as LifeMap, which sets out the stages 
through which students are expected to progress to achieve success. The goal is for faculty 
members to incorporate the success principles spelled out in LifeMap into their teaching, so that 
all developmental courses (not just specially designated Student Success courses) will help 
students acquire the academic and personal skills (note-taking, time management, and so on) 
that they will need throughout their college careers. Materials related to these concepts are being 
organized into a coherent program that will be available online, so that adjuncts as well as full- 
time staff will be readily able to participate. 

DEI funding also enabled participating colleges to leam from one another, as well as 
from colleges outside the DEI network. Thus, Guilford Tech adopted student tracking software 
that had been developed at Sinclair Community College. The grant also made funding available 
for site visits and for conference attendance. Faculty at Norwalk Community College, as well as 
a large contingent of faculty from other schools in the Connecticut system, attended a full-day 
workshop on reflective practice; the workshop emphasized critical thinking and integrative 
learning in learning communities and through electronic portfolios. Staff from Cuyahoga, North 
Central State, Sinclair, and Patrick Henry attended a conference in Baltimore on the Accelerated 
Learning Program (ALP) developed by the Community College of Baltimore County and went 
home determined to implement it on their own campuses. The DEI program director at Cuya- 
hoga met with Eastern Gateway personnel on three occasions to help them with the redesign of 
their developmental English program. And Zane State College hosted a daylong Ohio Commu- 
nity College Math Summit in which 22 of the state’s 23 community colleges participated. The 
meeting stimulated a good deal of discussion; breakout sessions were led primarily by faculty 
from the DEI colleges, who shared their innovative developmental math strategies and outcome 
data with persomiel from non-DEI and AtD institutions. 

Furthermore, at meetings convened by MDC and the Community College Leadership 
Program at the University of Texas, DEI program directors and other staff members at the 15 
colleges had opportunities to interact and to share their experiences with one another. 


Broader Campus Change 

Participation in the DEI brought new prominence to developmental education. A change in 
the organizational structure at Zane State College gave official recognition to the importance 
of the Developmental Education department. The department, which had previously been an 
academic support department with professional staff employees, was made a department 
within the Division of Arts and Sciences, and four full-time developmental education faculty 
positions were created. College administrators reported that integrating the department into 
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the academic division increased the credibility of the developmental education program, 
enhanced the role and title of the developmental education director (who was named an 
associate dean), and enabled the developmental instructors to become more fully engaged in 
the college’s academic processes. 

At Cuyahoga Community College, the DEI led to the institution of the Developmental 
Education Council, a large policymaking body that includes administrators, faculty, and staff. 
The group’s deliberations led to the implementation of a policy banning late course registration 
and a new test preparation intervention. 

The DEI prompted new thinking about developmental education. At Guilford Tech, for 
example, administrators who were interviewed in the third year of the grant were asking new 
questions about placement tests and about the very need for developmental education (at least 
for some students). As one commented, “We’ve learned through research that we are doing 
ourselves and our students a disservice by having everything hinge on a single placement test 
score. . . . We’ve even thought and talked about: If there are segments of students who are sort 
of borderline, why not let them go ahead and try [college- level courses], then remediate if it 
doesn’t work out?” 

The DEI director at Valencia spoke for many of his counterparts elsewhere in assert- 
ing that, in part because of the college’s participation in AtD and the DEI, a new recognition 
of the importance of ensuring college readiness and college success (not just access) has 
developed on campus. Faculty and staff, he noted, now see helping students develop success 
skills — once thought of as the province of the college’s student success course — as a 
responsibility that all share. 

Finally, at some institutions, the DEI was the impetus for a larger change in campus 
priorities. North Central State College provides a case in point. For this small college with 
limited institutional research (IR) capacity at the start of the grant, scaling up strategies while 
setting up the infrastructure to measure them was particularly challenging. A college adminis- 
trator advised that any small college embarking on an initiative like the DEI make sure that its 
information technology (IT) capacity be in place first, noting: 

It’s been a challenge for us. We’re a small institution, and here’s the advice I’d 
give to another small institution: Assess your IT and IR capacity. Use a logic 
model or similar tool to determine what you measure and why before you devel- 
op the strategy. 

On the other hand, she asserted that the dual challenges of reforming developmental education 
and developing strong data systems created institutionwide momentum and mobilized faculty 
and staff in pursuit of both goals: 
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The focus on our 1R capacity and our IT capacity became priorities for the col- 
lege and opportunities to bring together people who we needed to bring to the ta- 
ble. DEI changed us forever. 
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Chapter 5 

The DEI and Student Outcomes 


The preceding two chapters deal with the implementation of the Developmental Education 
Initiative (DEI); this chapter, in contrast, discusses the initiative’s outcomes. The data on 
student outcomes that the colleges submitted to JBL Associates provide preliminary and 
suggestive findings about the specific focal strategies that the colleges implemented and about 
whether the DEI as a whole made a difference for student success. 1 This chapter presents these 
findings and relates them to the existing literature about what works to improve the trajectories 
of developmental students. 

An important caution is in order, however: The findings cannot be taken as conclusive 
evidence about the impacts of the DEI on student outcomes. A rigorous impact study using 
random assignment or a strong alternative research design was not part of the charge to the DEI 
evaluators. Therefore, the methods available to the evaluators can show that the DEI was 
associated with the outcomes that were observed, but they cannot prove that the DEI caused 
these outcomes. 

The data are analyzed in two ways. First, outcomes for students who participated in the 
individual focal strategies are compared with outcomes for nonparticipants. Second, outcomes 
for cohorts of students who enrolled in the institutions after the DEI was put in place are 
compared with outcomes for earlier cohorts. 


Key Findings 

• When outcomes for participants in the focal strategies were compared with 
outcomes for nonparticipants, the majority of outcome differences (61 per- 
cent) were not statistically significant. Of those outcome differences that 
were statistically significant, a far higher percentage were positive than nega- 
tive (35 percent and 4 percent, respectively), suggesting that participation in 
the focal strategies benefited students rather than harmed them. 

• The DEI strategies were especially likely to be associated with increases in 
credits earned during the first term and with greater success in passing the 
gatekeeper English course by the end of the second term. 


'Chapter 2 describes the participating colleges (Tables 2.1a and 2.1b) and the focal strategies that each 
college adopted (Table 2.2). 
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• Strategies that involved contextualized instruction and collaborative learning 
were especially likely to be associated with positive outcome differences. 

• On the whole, students who enrolled in the colleges after the inception of the 
DEI (whether or not they participated in any of the focal strategies) achieved 
outcomes that are either better or no worse than outcomes of students who 
enrolled in the colleges earlier. 

• In general, strategies that reached more than 50 percent of the students whom 
they aimed to serve were more likely to be associated with positive outcomes 
than strategies that reached smaller numbers of students. 


Outcomes Associated with Participation in the Focal Strategies 

Die first set of analyses examines the outcomes associated with each of 31 focal strategies. 2 It 
examines students who were identified by the college as being in the target group for the strategy 
during their first tenn and compares outcomes for students who participated in the intervention 
during that tenn with outcomes for students in the target group who did not participate. 

Differences in outcomes between those two groups may be attributable to the interven- 
tion itself, but they could also be driven by preexisting differences between the two groups. 
Consider the case of a self-paced, computerized math course. Although the target group might 
include all students, such a course might well be more attractive to younger students, who are 
arguably more at ease with computers than their older counterparts; the latter might prefer more 
traditional instructional modalities. But younger students in the computerized classes, having 
attended high school more recently, may also remember the contents of their high school math 
courses much better than older students in traditional classes. If computerized classes have 
better outcomes than traditional classes, it would be inappropriate to conclude that these 
outcomes are due to the self-paced, technology-oriented nature of the courses, since the two 
kinds of classes serve different groups of students with different levels of skills at entry. The 
analyses employ regression adjustment to attempt to control for such initial differences using 
available data on measured student characteristics. 3 The simplicity of this approach makes it 


2 The strategies included are those for which data are available and for which there are sufficient numbers 
of students in both the treatment and the comparison group to allow a reliable estimate of differences between 
the groups. For more information, see Appendix A: Technical Appendix. 

3 The regression analysis controls for term of entry (combining the winter and spring terms into a single 
“spring” term for the three colleges on the quarter system), gender, age (whether the student was 25 or older or 
younger than 25), minority status, the level of each developmental class (if any) to which the student was 
referred in all three developmental subjects (math, reading, and writing), and whether the student was partici- 
pating in another focal strategy in that first term. 
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possible to apply a consistent analytic method across the many different types, time frames, and 
target populations of the focal strategies. 

The approach has decided limitations, however: Unmeasured characteristics between 
the groups cannot be controlled for, and these could have an important influence on program 
outcomes — potentially, at least as important or more important than the variables that can be 
measured and controlled for. As an example, consider a situation in which counselors encourage 
highly motivated students to take an accelerated course while encouraging less motivated 
students to take the normal developmental sequence. Because motivation is umneasured in this 
study (as in most studies), it cannot be included as a control variable in the analysis; as a result, 
the more motivated students participating in the accelerated course are likely to have stronger 
outcomes, regardless of whether the intervention itself is effective. As noted above, a random 
assignment design, which is the only way to control fully for umneasured variables, was not 
planned for this study. 

A further complication is that the accuracy and reliability of each estimate is dependent 
on the definition of the treatment and comparison groups created by each college. For these 
reasons, the results discussed here should not be interpreted as representative of true interven- 
tion impacts. 

The analyses examine five key variables: 

• Credits earned in the first term (including credits for all courses, whether de- 
velopmental or college-level, that were passed with a grade of D or better) 

• Grade point average (GPA) in the first term (including credits earned at a 
college from which the student transferred, if applicable) 

• Persistence (that is, whether the student returned to college) at the beginning 
of the second term 4 

• Whether students passed the “gatekeeper” course in English (that is, the first 
college-level English course required for completion of a degree) by the end 
of the second term 5 


4 For cohorts that entered in the fall, second-term outcomes include outcomes registered that fall or in the 
next winter-spring term; for cohorts that entered in the winter or spring, second-term outcomes include those 
registered during that term and in the following fall term. 

5 Outcomes for passing gatekeeper courses are calculated as unconditional outcomes; that is, if a student 
has not enrolled in the gatekeeper course or has enrolled but failed to pass the course, the student is counted as 
having “not yet passed” the course. 
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• Whether students passed the gatekeeper course in math (the first college- 
level math course required for completion of a degree) by the end of the se- 
cond term 6 

Of the 3 1 strategies, all had sufficient follow-up data to examine first-term outcomes (credits 
and GPA) but not necessarily second-term outcomes. 7 Moreover, English and math gatekeeper 
outcomes were calculated only for strategies for which these outcomes were relevant. (For 
example, math outcomes were not calculated for English-oriented strategies.) Thus, the number 
of strategies for which second-term outcomes are measured is less than the number of strategies 
for first-term outcomes. 

Overall Findings 

Table 5.1 shows the percentage of focal strategies for which outcome differences be- 
tween participants and nonparticipants are significantly positive (that is, participants had better 
outcomes than nonparticipants), significantly negative (participants had worse outcomes than 
nonparticipants), or neutral (participants’ outcomes do not differ significantly from outcomes of 
nonparticipants). 8 

The most important finding appears in the last row of the table. The data indicate that 
while the majority of strategies (59 percent) show neither positive nor negative outcome 
differences, of the remaining strategies, far more appear to be associated with positive than with 
negative outcome differences (36 percent and 5 percent, respectively). The DEI strategies were 
especially likely to be associated with increases in credits earned during the first term and with 
greater success in passing the gatekeeper English course by the end of the second term. 

Findings by Strategy Objective 

Table 5.2 shows the number of focal strategies for which first-term outcome differences 
between participants and nonparticipants are positive, neutral, or negative, broken out by 


6 ln many colleges, several different math courses may fulfill the gatekeeper requirement; if the student 
passed any one of those courses, the student is considered to have fulfilled the requirement. 

7 While term 1 outcomes include all cohorts, term 2 outcomes exclude students entering in fall 2011 (due 
to lack of follow-up data). In addition, the term 2 outcomes were all analyzed using logistic regression, which 
has additional sample-size requirements (discussed in Appendix A); some strategies could not meet these 
requirements and were excluded from term 2 analyses for that reason. 

8 The terms “significance,” “significant,” and “significantly” are used in this report to refer to the concept 
of “statistical significance,” that is, the likelihood that a difference arose by chance. Differences in this report 
are considered to be statistically significant if the probability that they arose by chance is less than 5 percent. 
Table 5.1 presents summary data. For detailed information on the coefficients and standard errors in the 
regression models, please see Appendix A. 
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The Developmental Education Initiative 
Table 5.1 

Outcome Differences Associated with Participation 
in the DEI Focal Strategies 

Number of 
Strategies 


Percentage of Focal Strategies for for Which 

Outcome Which Outcome Difference Is: Outcome Was 


Measured 

Positive 

Neutral 

Negative 

Total 

Measured 

First-term outcomes 






Credits earned 
in first term 

45.2 

45.2 

9.6 

100.0 

31 

GPA at end of 
first term 

29.0 

67.8 

3.2 

100.0 

31 

Second-term outcomes 






Persistence to 
beginning of 
second term 

28.6 

67.9 

3.5 

100.0 

28 

Passed gatekeeper 
English by end of 
second term 

50.0 

50.0 


100.0 

18 

Passed gatekeeper 
math by end of 
second term 

29.4 

64.7 

5.9 

100.0 

17 

All outcomes 

36.0 

59.2 

4.8 

100.0 

125 a 


NOTES: The table shows outcomes associated with participation in the focal strategies; it cannot be 
inferred that participation caused these outcomes. 

a This figure represents the total number of times that the 3 1 focal strategies were counted in 
measuring the outcome differences for all first-term and second-term outcomes considered together. 


strategy objective. Table 5.3 presents parallel information for outcomes measured in the second 
term. Finally, Figure 5.1 summarizes the findings in graphic form. 

The acceleration- and avoidance-oriented strategies were associated with outcomes that 
either are significantly better for participants than nonparticipants or, more commonly, are 
similar for the two groups. The few negative differences were all associated with strategies that 
fall under the rubric of support. Strategies that fall under the objective of instructional relevance 
— that is, strategies centered on contextualized instruction and collaborative learning — were 
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The Developmental Education Initiative 
Table 5.2 


First-Term Outcome Differences Associated with Participation in the 
DEI Focal Strategies, by Strategy Objective 


Outcome 

Measured 

and 

Strategy 

Objective 

Percentage of Focal Strategies for 
Which Outcome Difference Is: 
Positive Neutral Negative 

Total 

Number of 
Strategies 
for Which 
Outcome Was 
Measured 

Credits earned in term 






Accleration 

36.4 

63.6 

- 

100.0 

11 

Avoidance 

66.7 

33.3 

- 

100.0 

3 

Instructional 






relevance 

66.7 

33.3 

- 

100.0 

3 

Support 

42.9 

35.7 

21.4 

100.0 

14 

All objectives 

45.2 

45.2 

9.6 

100.0 

31 

GPA at end of term 






Accleration 

18.2 

81.8 

- 

100.0 

11 

Avoidance 

33.3 

66.7 

- 

100.0 

3 

Instructional 






relevance 

66.7 

33.3 

- 

100.0 

3 

Support 

28.6 

64.3 

7.1 

100.0 

14 

All objectives 

29.0 

67.8 

3.2 

100.0 

31 


NOTE: The table shows outcomes associated with participation in the focal strategies; it cannot be 
inferred that participation caused these outcomes. 


especially likely to be associated with statistically significant and positive outcome differences. 
There are only three examples of such strategies, and it is risky to generalize from such a small 
sample. More rigorous studies confirm, however, that contextualized instruction in vocational 
programs and learning communities has helped students earn more course credits and progress 
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The Developmental Education Initiative 
Table 5.3 


Second-Term Outcome Differences Associated with Participation in the 
DEI Focal Strategies, by Strategy Objective 


Outcome 

Measured 

and 

Strategy 

Percentage of Focal Strategies for 
Which Outcome Difference Is: 


Number of 
Strategies 
for Which 
Outcome Was 

Objective 

Positive 

Neutral 

Negative 

Total 

Measured 

Persistence to beginning of second term 





Accleration 

- 

100.0 

- 

100.0 

9 

Avoidance 

66.7 

33.3 

- 

100.0 

3 

Instructional 

relevance 

66.7 

33.3 

— 

100.0 

3 

Support 

30.8 

61.5 

7.7 

100.0 

13 

All objectives 

28.6 

67.9 

3.5 

100.0 

28 

Passed gatekeeper English bv end of second term 




Accleration 

100.0 

- 

- 

100.0 

3 

Avoidance 

- 

100.0 

- 

100.0 

2 

Instructional 

relevance 

66.7 

33.3 

— 

100.0 

3 

Support 

40.0 

60.0 

- 

100.0 

10 

All objectives 

50.0 

50.0 

- 

100.0 

18 

Passed gatekeeper math bv end of second term 




Accleration 

25.0 

75.0 

- 

100.0 

4 

Avoidance 

- 

100.0 

- 

100.0 

3 

Instructional 

relevance 

100.0 

— 

— 

100.0 

1 

Support 

33.3 

55.6 

11.1 

100.0 

9 

All objectives 

29.4 

64.7 

5.9 

100.0 

17 


NOTE: The table shows outcomes associated with participation in the focal strategies; it cannot be 
inferred that participation caused these outcomes. 
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Percentage of outcome differences 


The Developmental Education Initiative 
Figure 5.1 


Outcome Differences Associated with Participation in the DEI Focal Strategies, 
by Strategy Objective and Whether Difference Is Positive, Neutral, or Negative 
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Acceleration Avoidance Instructional relevance Support 

(N = 38) (N = 14) (N = 13) (N = 60) 

Strategy objective 


■ Positive 
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□ Negative 


NOTE: The figure shows outcomes associated with participation in the focal strategies; it cannot be inferred that participation 
caused these outcomes. 



from developmental into college-level coursework. 9 


Degree of Scale-Up and Differences in Outcomes 

In general, as Chapter 3 documents, most of the focal strategies affected a modest per- 
centage of students to whom they were targeted. It is reasonable to ask whether strategies that 
were scaled up to a greater extent also were more likely to be associated with statistically 
significant and positive differences between outcomes for participants and nonparticipants. 

As in Chapter 3, two measures of scale-up were employed: (1) the number of students 
participating in each focal strategy as a percentage of the college’s goal for that strategy during the 
2010-201 1 academic year and (2) the number of students participating in each focal strategy as a 
percentage of the strategy’s target population in fall 2011. Along both measures, each focal 
strategy was sorted into one of three categories by extent of scale-up (whether the strategy reached 
0 to 24 percent, 25 percent to 50 percent, or more than 50 percent of the goal or of the target 
populations) and into one of three categories by the valence of the outcome differences (positive, 
neutral, or negative). The results are presented in Appendix Table A.3 and Appendix Table A.4. 

In general, strategies that reached more than 50 percent of their objectives were more 
likely to be associated with positive outcomes than strategies that reached smaller numbers of 
students. What is less clear is the reason for this finding. It may be that colleges that were more 
effective in setting realistic goals for their strategies and in meeting them were also more likely 
to take steps to ensure that their strategies were effective. Aside from this “top tier” of strategies, 
there is no evidence that strategies that reached between 25 percent and 50 percent of targeted 
students were associated with better outcomes than those that reached less than 25 percent of 
their target populations. 


9 See Zachry Rutschow and Schneider, 2011. Their review of the research literature identifies placing de- 
velopmental students into college-level courses with additional supports and providing modularized or 
compressed courses as other approaches that, according to relatively rigorous studies, have shown the greatest 
benefits for developmental students. For a review of the literature on contextualized instruction, see Perin 
(2011); for findings on structured forms of student collaboration, see Hodara (2011). 

Because the acceleration and the support categories each include a diverse set of approaches, each catego- 
ry was examined more closely. Data presented in Appendix Table A.2 indicate that, among acceleration 
strategies, those involving modular strategies most commonly showed neutral differences, while the differ- 
ences among paired strategies were neutral in approximately half the instances and positive in the remainder of 
the instances. Results for the support strategies, shown in the same table, suggest that the strategies focusing on 
study skills and mentoring showed a mix of positive, negative, and neutral differences, while approaches 
focusing on tutoring and supplemental instruction showed no negative differences. Because the number of 
examples of each specific strategy is small, these results should be regarded with a good deal of caution. 
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Changes in Student Outcomes Over Time 

As Chapter 4 makes clear, the DEI includes programs and policies other than those embodied in 
the focal strategies; arguably, then, students might benefit from the initiative whether they 
participated in the focal strategies or not. The next set of analyses addresses this question more 
directly, by tracking changes in students’ outcomes across time, comparing average outcomes 
for cohorts of students who entered the participating colleges before and after the DEI was put 
into place. 10 As with the findings for the individual focal strategies, any differences in the 
outcomes for pre- and post-DEI cohorts cannot be taken as definitive evidence of the DEI’s 
effectiveness (or lack of effectiveness). These differences could arise because of changes in the 
skill levels of students (if, for example, changes in labor market conditions induced better- or 
less-educated students to enroll). As with the analysis of outcomes associated with the strate- 
gies, the analysis excludes this alternative explanation for outcome differences to the extent 
possible by controlling for measured student characteristics, but it cannot control for unmeas- 
ured characteristics. 11 In addition, other changes in the colleges that occurred over the same time 
period but were completely unrelated to the DEI might also have helped to produce differences. 
At best, the findings of this analysis should again be regarded as suggestive of the DEI’s 
potential to make a difference for students, but they are far from definitive. 

Table 5.4 shows the results for the same five outcomes that are examined in the section 
above. In the table, a plus sign (+) indicates that the outcome level is significantly higher after 
the DEI; a minus sign (-) indicates that the average outcome level is significantly lower after the 
DEI; and a zero (0) indicates that the difference is not large enough to be statistically significant 
in either direction. 12 

First, across the 75 outcomes presented in the table (5 outcomes for each of the 15 col- 
leges), 28 outcomes (37 percent) are significantly positive; 20 outcomes (27 percent) are 
significantly negative; and 27 outcomes (36 percent) are neither positive nor negative. In other 
words, on the whole, the post-DEI cohort achieved outcomes that are either significantly better 
or no worse than the outcomes of their pre-DEI counterparts. 


10 The pre-DEI cohorts include students first enrolled between fall 2002 and fall 2009. The post-DEI cohorts 
include students first enrolled between winter or spring of the 2009-2010 academic year and fall 2011 (regardless 
of whether the student participated in any DEI focal strategies). Defining the DEI as beginning in winter or spring 
2010 gives the colleges a “grace period” of several months for planning and piloting interventions. 

11 The measured characteristics that were controlled for include entry in the fall versus the spring term, 
gender, age, minority status, and level of developmental referral for each of the three developmental subjects. 
For more information about the analysis, see Appendix A. 

12 The post-DEI cohort for second-term outcomes excludes students who enrolled in fall 2011, since two 
terms of follow-up are not available for this group. 
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The Developmental Education Initiative 
Table 5.4 


Outcomes of Post-DEI Cohorts Compared with 
Outcomes of Pre-DEI Cohorts 


College 

Credits 
Earned 
in First Term 

GPA in 
First Term 

Persistence 
into Second 
Term 

Passed 
Gatekeeper 
English by 
End of 
Second Term 

Passed 
Gatekeeper 
Math by 
End of 
Second Term 

A 

+ 

+ 




B 

+ 

+ 

0 

0 

0 

C 

+ 

- 

+ 

+ 

- 

D 

+ 

+ 

+ 

+ 

+ 

E 

- 

0 

0 

+ 

+ 

F 

0 

- 

+ 

0 

- 

G 

0 

0 

+ 

0 

0 

H 

+ 

- 

0 

0 

0 

I 

0 

- 

0 

+ 

+ 

J 

0 

0 

0 

+ 

0 

K 

0 

- 

+ 

+ 

- 

L 

- 

- 

+ 

- 

- 

M 

- 

- 

- 

- 

0 

N 

+ 

+ 

0 

0 

0 

O 

+ 

0 

0 

+ 

+ 


NOTES: The table shows outcomes associated with belonging to a cohort of students entering 
college after the winter-spring term of the 2009-2010 academic year, when the DEI began; it 
cannot be inferred that the DEI caused these outcomes. 

A plus sign (+) indicates that the average value of the outcome for students entering the 
college during or after the winter-spring term of the 2009-2010 academic year is greater than for 
students entering the college between fall 2002 and fall 2009 and that the difference is 
statistically significant at a level less than or equal to 5 percent. 

A minus sign (-) indicates that the average value of the outcome for students entering the 
college during or after the winter-spring term of the 2009-2010 academic year is less than for 
students entering the college between fall 2002 and fall 2009 and that the difference is 
statistically significant at a level less than or equal to 5 percent. 

A zero (0) indicates that the average value of the outcome for students entering the college 
during or after the winter-spring term of the 2009-2010 academic year does not differ 
significantly from that for students entering the college between fall 2002 and fall 2009. 


Compared with students entering the colleges before the DEI was put in place, post- 
DEI cohorts were more likely to register statistically significant improvements on three out- 
comes: credits earned during their first term, persistence into the second term, and passing their 
college’s gatekeeper English course by that term’s end. Post-DEI entrants, however, were more 
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likely to have lower first-term GPAs than their pre-DEI counterparts. A possible explanation is 
that students who were taking accelerated courses or who were enrolled in learning communi- 
ties faced more challenging work and achieved lower (though still passing) grades than pre-DEI 
students who were enrolled in standard developmental classes. Post-DEI students were also less 
likely to pass the gatekeeper course in math by the end of their second term. While this finding 
is not definitive, it merits further exploration, given the number of colleges that implemented 
modularized and/or computerized self-paced math as one of their focal strategies. 13 

At 6 of the 15 colleges, post-DEI students did either better or the same as earlier cohorts 
on the five outcomes that were measured, and at one of these colleges — Guilford Tech — 
students did significantly better than their pre-DEI counterparts on all five outcomes. At the 
remaining 9 colleges, post-DEI students gained ground on some outcomes relative to their pre- 
DEI peers but lost ground on other outcomes. 


13 Of the four colleges that implemented a modularized and/or computerized math strategy and had suffi- 
cient data to perform comparisons, only one showed positive outcome differences. In contrast, English 
acceleration strategies tended to focus on paired courses; all three of these strategies with sufficient data to 
perform comparisons exhibited positive outcome differences. 
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Chapter 6 

The DEI in Broader Perspective 


The title of MDRC’s interim report on the Developmental Education Initiative (DEI), Scaling Up 
Is Hard to Do, pithily summarizes that report’s message in covering the first year of the initia- 
tive. 1 In the subsequent two years, the efforts of the colleges that participated in the demonstra- 
tion have contributed important lessons to the relatively young field of scaling initiatives in 
higher education. The present report points to the salience of resources, communications, 
engagement, and a commitment to uniform instructional practice for scaling efforts. It also 
indicates that the DEI was the springboard for other discussions and reforms at the colleges. 

Two years later, however, the message of the earlier report remains unchanged: Scaling 
up is hard to do. The findings suggest that the proportion of incoming students touched by any 
of the colleges’ focal strategies doubled between fall 2009 and fall 2011. Still, most strategies 
fell considerably short of the numerical targets that the colleges had set for themselves, and 
most reached under half the students in their target populations at the demonstration’s end. 

The preceding chapter suggests that a number of the strategies (especially ones involv- 
ing contextualization of instruction and collaborative learning) are associated with positive and 
statistically significant differences in student outcomes. While this evidence is promising, it 
cannot be taken as definitive because the analysis does not allow for causal conclusions; that is, 
one cannot infer that the strategies caused the positive outcomes. A much smaller number of 
strategies appears to be associated with significantly worse outcomes for students, and the 
majority of the strategies do not appear to be associated with either better or worse outcomes — 
although, again, causal inferences cannot be drawn. 

On the whole, from the quantitative perspective, the DEI represents a modest improve- 
ment for the participating colleges. In some ways, this is not surprising. First, the initiative was 
about the scaling-up of specific strategies, not about a sweeping reform of developmental 
education, which might have produced stronger results. Second, the meaning of “scaling up” 
was largely left undefined. Had colleges been instructed at the outset to mount interventions that 
would reach very large proportions of developmental students, they might have elected to 
expand different strategies. 

Third, while colleges were instructed to scale up effective practices, at the inception of 
the demonstration there was little robust evidence about just which practices are effective, to 
inform their choices about what to scale. Instead, colleges made decisions, at best, on the basis 

'See Quint et al. (2011). 
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of data that seemed to indicate that a particular strategy was associated with better outcomes 
for students. Only rarely were the findings grounded in strong enough research designs to 
conclude that the strategy had caused those outcomes to occur. A theory of change that “more 
is better” — the theory that essentially undergirded the DEI — cannot be expected to benefit 
students if the intervention being scaled up has not been shown through rigorous research to 
produce positive impacts in the first place. Unfortunately, the demonstration did not build in 
opportunities to solidify the evidentiary base. Thus, although the information in Chapter 5 
suggests that some focal strategies fostered students’ progress and achievement more than 
others, it, too, is inconclusive. 

Fourth, as Chapter 3 details, while adequate resources (including personnel, space, and 
materials), good communication (directed toward both staff and students), and commitment to a 
uniform mode of instruction facilitated scaling up, expansion was impeded when any of these 
factors was absent or in short supply. And even when the benefits of a particular strategy were 
explained to them, students could not always be persuaded to participate. Indeed, the nature of 
the student population at community colleges — more often than not part-time, and often 
juggling work and family responsibilities — is in itself an obstacle to scaling up, when the 
activities being scaled up are a matter of choice and when they are available only at select times 
and in specific locales that do not meet students’ needs. (Online instruction may be a partial 
solution to this problem, but it is unlikely to be appropriate for all students.) 2 

Finally, expanding many strategies takes time. This could be true as well of instruction- 
al strategies scaled up through “gentle fiat,” a process that essentially eliminated choice about 
how instructors would teach their classes. In one case where strategies were expanded in this 
way, a yearlong planning period was involved. When scale-up involves strategies that are 
voluntary rather than mandated and that, therefore, involve proselytization and eventual conver- 
sion of initially agnostic instructional staff, the time required for full scale-up is probably longer. 

In this regard, it is pertinent to inquire into the future of the DEI’s innovations. And it 
appears that, for several reasons, the DEI will leave a lasting legacy at the participating colleges. 

First, many of the focal strategies that were inaugurated or expanded under the DEI will 
be “institutionalized” — supported with regular college funding rather than grant dollars. At 
some colleges, leaders were committed from the start to developing or expanding strategies that 
they intended to sustain. As the president of Danville Community College put it: 

My belief is that if you accept money from a foundation, you also should have in 
your college plan a commitment to the extent that you can to ensure that the 
broad goals of that grant continue, to ensure sustainability. When you accept 


2 With regard to online instruction, see Jaggars (2012). 
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grant money, there’s an assumption that you’re really serious about integrating 
this into the culture of the college, the way we do things. We’re not backing off 
of this, because we really believe in this. 

Other institutions did not share this level of advance planning and commitment but 
nonetheless had decided to move forward with strategies that they deemed successful, especial- 
ly if the strategies entailed redeployment of preexisting staff rather than the continued employ- 
ment of new staff. (At a couple of institutions, positions of case managers who had been 
specifically hired for the DEI were especially at risk.) 

Second, some lessons learned through the DEI will carry over into future initiatives. In 
this regard, it is significant that two DEI colleges — Guilford Technical Community College 
and Sinclair Community College — are participating in the Bill & Melinda Gates Foundation’s 
new initiative, known as Completion by Design (CBD), which aims at restructuring the entire 
community college experience. These colleges are serving as the lead institutions in their states’ 
“cadres” of colleges participating in CBD. Interventions mounted at these colleges as part of the 
DEI (among them, computerized and modularized math classes and mandatory orientation) will 
be implemented as part of CBD in both these lead colleges and other colleges in the cadres. 

Finally, at some campuses, leaders entrusted responsibility for the DEI, and thereby 
brought increased visibility, to a new generation of talented and hard-working administrators. 
As these administrators move into new positions — two of them as heads of multi-college CBD 
cadres — they are likely to carry into new arenas DEI’s message about the importance of 
student success. As the president of Sinclair Community College explained: 

It has been vital to have an organization full of people engaged both individually 
and collectively in learning, especially when [because of retirements] half of the 
faculty is new. DEI has fostered a learning culture that has forced us to think 
about our mission, values, and principles. ... It keeps our eye on the fact that 
low-income populations need educational access and equity. 

In summary, the Developmental Education Initiative has infonned the field of develop- 
mental education, resulting in much-needed conversations about what scaling means and about 
what is required for successful scale-up. At the colleges, the DEI has deepened and expanded the 
interventions and planted seeds for further change. At the same time, the results suggest caution 
about the speed with which community colleges can meet highly ambitious goals, when less 
ambitious objectives require time, resources, communication, engagement, and commitment. 
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Appendix A 

Technical Appendix 




The DEI Data Set 

All Achieving the Dream (AtD) institutions submit anonymized data on individual students’ 
characteristics and outcomes to a data set managed by JBL Associates. The AtD data set — 
which has been used in numerous publications to document rates of developmental student 
progression and success 1 — includes demographic infonnation, placement referral levels upon 
college entry, and tenn-by-tenn infonnation on each student’s cumulative grade point average 
(GPA) as well as developmental and gatekeeper course enrollment and performance. Round 1 
and Round 2 AtD schools submitted data on cohorts of students entering the college in fall 2002 
and in every fall thereafter. An AtD “cohort” includes all degree- or certificate-seeking under- 
graduate students (full- and part-time) entering the institution and attempting credits for the first 
time during the fall term. Students must be enrolled in courses creditable toward a degree, 
diploma, certificate, or other formal award, including vocational, occupational, or distance 
learning programs. Dual-enrolled high school students are excluded. 2 All students are followed 
longitudinally, with follow-up data for each cohort being submitted annually. 

When they began the DEI, then, the colleges had already submitted data on all cohorts 
from fall 2002 to fall 2008. The requirements for the DEI database were expanded to allow for 
more precise tracking over the span of the three-year initiative, in two key ways. First, colleges 
were required to submit data not just on fall cohorts but also on entering winter/spring cohorts, 
beginning with winter/spring 2008. Second, colleges were required to identify a specific target 
population for each of their focal strategies. Then, separately for each strategy, the college was 
instructed to indicate on a tenn-by-tenn basis whether each targeted student was touched by the 
strategy (the treatment group) or not (the comparison group). Students who were not in the 
target group for a given strategy in a given tenn would have blank participation data for that 
strategy in that tenn. 

As noted in Chapter 2, each strategy within each college could have a quite different 
target group. Some strategies targeted developmental math students while others targeted 
developmental English students; some targeted higher-level developmental students while 
others targeted lower-level students; some strategies targeted only first-tenn students while 
others targeted a mix of first-term and continuing students; some strategies targeted a mix of 
developmental and college-ready students; and several strategies included other “at-risk” 
criteria (not included in the DEI database) to determine eligibility for the target population. For 
some colleges, the defined target group shifted over time, as the college refined its strategy, or 
its understanding of which populations might benefit most strongly from the program. Within 


’Bailey, Jeong, and Cho (2010). 

2 The cohort does, however, include students who have prior high school credit or who have recently grad- 
uated from a dual enrollment program and are now college students. 
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the target population for the given term, each college also had latitude to determine whether 
each student was treated or untreated. For instructional strategies, the definition for the treat- 
ment and comparison groups was typically clear: “treated” students were targeted students who 
took strategy-based sections of a given course or curriculum, and “comparison” students were 
targeted students who took alternate versions of the curriculum. For support strategies, it was 
often less clear how to define the treatment and comparison. For example, for a tutoring 
program, one college was unsure whether to define a “treated” student as one who attended 
tutoring just once or on a more regular basis; the college eventually decided to define the 
tutoring treatment as receiving at least one hour of tutoring during the given quarter. As another 
example, a multicampus school initially rolled out a case management program at a subset of its 
campuses, and it initially defined each target student on one of those campuses as receiving the 
case management treatment, whether the student was contacted by a case manager or not. In 
later semesters, the college reconceptualized the treatment group as those students who received 
actual contact from a case manager. 

For some strategies, treatment data were difficult to track in some semesters. For exam- 
ple, one program coordinator explained that in the first term of the treatment strategy, the 
support service data were marked down on spreadsheets by individual faculty and staff mem- 
bers; those sheets were inconsistently stored, collected, and entered into the system, and thus the 
true treatment numbers for that term are unknown. Such instances are documented in Appendix 
Table A.l with the code “UNK” under the appropriate percentage column. 

In other cases, the treatment data were collected regularly and consistently, but no com- 
parison students were marked in the database for some or all temis. In those cases, the authors 
contacted each college and determined whether (1) each member of the target group was, 
indeed, served by the strategy during the terms in question — that is, no comparison students 
existed; or (2) the college simply did not flag the comparison students during the tenns in 
question. In the latter case, the authors used the college’s definition of the target group (for 
example, “developmental English students”) to construct an appropriate comparison group 
within the DEI data set, flagging students who were in the strategy’s target group during the 
given semester but who had not yet been not treated by the strategy. 

In some cases, it was not possible to construct an appropriate comparison group, be- 
cause criteria necessary to define the target group were not available in the DEI data set. In 
these cases, the degree of scale for the strategy in the given semester was unknown. Such cases 
are documented in Appendix Table A.l with the code “UNK” under the appropriate percentage 
column; those strategies are omitted from Appendix Table A.2. 
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The Developmental Education Initiative 
Appendix Table A.l 


Percentage of Target Population Served by Each Focal Strategy Each Fall and Spring 
Term Between Fall 2009 and Fall 2011, and Strategy Characteristics 


Objective 

Approach 

College 

New or 
Scaled Up 

Percentage of Target Population Served This Term 
Fall '09 Spring TO Fall TO Spring 'll Fall'll 

Acceleration 11 

Modular 

College F 

N 

0 

0 

0 

0 

87 

Acceleration 1 

Modular 

College F 

N 

0 

0 

96 

91 

91 

Acceleration 

Modular 

College L 

S 

<1 

<1 

<1 

<1 

1 

Acceleration 1 

Modular 

College I 

N 

0 

0 

1 

11 

9 

Acceleration 

Modular 

College G 

N 

0 

36 

9 

82 

34 

Acceleration 1 

Modular 

College J 

N 

0 

0 

0 

59 

30 

Acceleration' 1 

Modular 

College M 

N 

15 

29 

24 

17 

UNK b 

Acceleration 11 

Modular 

College K 

N 

3 

8 

11 

35 

32 

Acceleration 

Modular 

College A 

S 

2 

17 

14 

25 

27 

Acceleration' 1 

Modular 

College A 

S 

3 

7 

8 

10 

11 

Acceleration 

Modular 

College A 

N 

8 

36 

33 

25 

55 

Acceleration' 1 

Paired 

College J 

N 

5 

53 

13 

65 

11 

Acceleration 

Paired 

College N 

S 

9 

12 

13 

15 

16 

Acceleration 

Paired 

College 1 

N 

0 

4 

3 

5 

6 

Avoidance 

Test prep 

College M 

N 

UNK C 

UNK b 

UNK C 

1 

UNK b 

Avoidance 3 

Test prep 

College K 

S 

5 

11 

6 

8 

3 

Avoidance 

Test prep 

College H 

S 

24 

27 

19 

11 

20 

Avoidance 

Test prep 

College D 

N 

0 

0 

31 

19 

26 

Instructional relevance 

Active learning 

College N 

S 

29 

61 

76 

62 

71 

Instructional relevance 

Contextualization 

College 0 

N 

94 

UNK C 

94 

93 

97 

Instructional relevance 

Contextualization 

College G 

N 

0 

44 

8 

70 

17 

Support 

Advising 

College J 

N 

UNK b 

UNK b 

UNK b 

UNK b 

66 

Support 

Advising 

College E 

S 

4 

6 

8 

9 

6 

Support 

Advising 

College K 

N 

0 

0 

0 

UNK C 

UNK C 

Support 

Advising 

College D 

S 

41 

28 

42 

54 

69 

Support 

Advising 

College D 

N 

0 

26 

16 

15 

0 
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Appendix Table A.l (continued) 


Objective 

Approach 

College 

New or 
Scaled Up 

Percentage of Target Population Served This Term 
Fall '09 Spring TO Fall TO Spring 'll Fall'll 

Support 

Advising 

College N 

S 

6 

7 

12 

11 

17 

Support 

Advising 

College G 

S 

0 

100 

100 

99 

100 

Support 

Advising 

College 0 

s 

46 

56 

83 

81 

77 

Support 

Advising 

College 0 

s 

12 

32 

41 

60 

42 

Support 

Study skills 

College B 

s 

15 

UNK b 

20 

23 

29 

Support" 

Study skills 

College C 

s 

4 

5 

5 

6 

5 

Support 

Study skills 

College E 

s 

6 

5 

12 

8 

4 

Support 

Study skills 

College L 

s 

0 

0 

0 

1 

1 

Support 

Study skills 

College L 

s 

4 

UNK b 

4 

7 

25 

Support 

Tutoring 

College E 

s 

0 

27 

UNK C 

47 

27 

Support 

Tutoring 

College H 

s 

2 

3 

3 

3 

2 

Support 

Tutoring 

College H 

s 

2 

3 

4 

4 

3 

Support" 

Tutoring 

College C 

s 

21 

40 

26 

41 

30 

High school support 

NA 

College I 

N 

0 

0 

27 

6 

86 

High school support 

NA 

College C 

s 

UNK C 

UNK C 

UNK C 

UNK C 

UNK C 

NA 

NA 

College J 

N 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

College M 

N 

14 

100 

UNK b 

100 

100 

NA 

NA 

College F 

N 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

College F 

N 

NA 

NA 

NA 

NA 

NA 

NA 

NA 

College H 

N 

NA 

NA 

NA 

NA 

NA 


SOURCE: MDRC and CCRC calculations from colleges' reports to JBL. 

NOTES: UNK = Unknown. 

NA = Not available. 

The table includes N = 46 focal strategies. Grayed-out rows (N = 7) are excluded from all Chapter 3 tables and figures because they could 
not be classified according to approach. Additional strategies had insufficient data for inclusion in Chapter 3 tables and figures (due to 
unknown percentages in either fall 2009 or fall 2011), resulting in N = 35 strategies. 

a The comparison group was constructed from the data, based on the school’s definition of the target group. 
b The sample size was not known for this term. 




The Developmental Education Initiative 
Appendix Table A.2 


Differences in Outcomes Between Targeted Students Who Participated and 
Who Did Not Participate in Each Focal Strategy During Their First Term 


College 

Objective 

Approach 

Credits 
Earned 
Term 1 

GPA 
Term 1 

Persist 
to Term 2 

Pass English 
Gatekeeper 
by Term 2 

Pass Math 
Gatekeeper 
by Term 2 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

College F 

Acceleration 2 

Modular 

IN 

IN 

IN 

IN 

NA 

College F 

Acceleration 2 

Modular 

IN 

IN 

IN 

NA 

IN 

College L 

Acceleration 

Modular 

IN 

IN 

IN 

NA 

IN 

College I 

Acceleration 2 

Modular 

-0.62 

-0.14 

0.24 

NA 

-0.31 




(0.45) 

(0.10) 

(0.24) 


(0.38) 

College G 

Acceleration 

Modular 

0.85 

-0.09 

-1.04 

NA 

0.54 




(1.58) 

(0.33) 

(0.93) 


(0.72) 

College J 

Acceleration 2 

Modular 

-2.11 

0.10 

IN 

IN 

NA 




(1.08) 

(0.34) 




College M 

Acceleration 2 

Modular 

-0.39 

-0.04 

-0.05 

NA 

IN 




(0.56) 

(0.14) 

(0.26) 



College K 

Acceleration 2 

Modular 

0.09 

0.01 

0.05 

NA 

-0.36 




(0.14) 

(0.05) 

(0.10) 


(0.18) 

College A 

Acceleration 

Modular 

0.62* 

0.21* 

0.09 

NA 

IN 




(0.26) 

(0.09) 

(0.25) 
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Appendix Table A.2 (continued) 


College 

Objective 

Approach 

Credits 
Earned 
Term 1 

GPA 
Term 1 

Persist 
to Term 2 

Pass English 
Gatekeeper 
by Term 2 

Pass Math 
Gatekeeper 
by Term 2 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

College A 

Acceleration 11 

Modular 

0.84* 

0.14 

0.27 

NA 

0.77* 




(0.34) 

(0.13) 

(0.28) 


(0.37) 

College A 

Acceleration 

Modular 

0.49 

0.01 

IN 

IN 

NA 




(0.49) 

( 0 . 11 ) 




College J 

Acceleration 11 

Paired 

1.36* 

0.45** 

-0.28 

1 73 *** 

NA 




(0.56) 

(0.17) 

(0.30) 

(0.29) 


College N 

Acceleration 

Paired 

1.63*** 

0.00 

0.32 

0.47* 

NA 




(0.46) 

(0.14) 

(0.24) 

(0.23) 


College I 

Acceleration 

Paired 

-0.06 

0.04 

-0.13 

1 21 *** 

NA 




(0.61) 

(0.14) 

(0.27) 

(0.28) 


College M 

Avoidance 

Test prep 

IN 

IN 

IN 

IN 

IN 

College K 

Avoidance 3 

Test prep 

0.56** 

0.34*** 

0.80*** 

0.21 

- 0.10 




(0.18) 

(0.06) 

( 0 . 12 ) 

( 0 . 11 ) 

( 0 . 20 ) 

College H 

Avoidance 

Test prep 

0.41 

0.16 

0.27 

NA 

-0.39 




(0.69) 

(0.14) 

(0.36) 


(0.44) 

College D 

Avoidance 

Test prep 

0.55* 

- 0.02 

0.30* 

0.22 

0.50 




(0.23) 

(0.07) 

(0.14) 

(0.17) 

(0.28) 

College N 

Instructional 

Active 

2 93*** 

0.33*** 

0 22*** 

0 7Q*** 

0 77 *** 


relevance 

learning 

(0.23) 

(0.07) 

( 0 . 11 ) 

(0.13) 

(0.16) 


(continued) 


Appendix Table A.2 (continued) 





Credits 
Earned 
Term 1 

GPA 
Term 1 

Persist 
to Term 2 

Pass English 
Gatekeeper 
by Term 2 

Pass Math 
Gatekeeper 
by Term 2 

College 

Objective 

Approach 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

College 0 

Instructional 

relevance 

Contextualization 

3 29*** 
(0.26) 

0.61*** 

(0.09) 

j jj*** 

(0.17) 

\ 77 ** 

(0.59) 

NA 

College G 

Instructional 

relevance 

Contextualization 

0.07 

(1.38) 

0.01 

(0.29) 

-0.43 

( 0 . 86 ) 

-0.83 

(0.74) 

NA 

College J 

Support 

Advising 

IN 

IN 

IN 

IN 

IN 

College E 

Support 

Advising 

1 37 *** 
(0.17) 

0.10* 

(0.05) 

0 32 *** 
(0.08) 

0.46*** 

(0.10) 

-0.15 

(0.28) 

College K 

Support 

Advising 

IN 

IN 

IN 

IN 

IN 

College D 

Support 

Advising 

0 7i*** 

(0.14) 

-0.04 

(0.04) 

0.20** 

(0.08) 

Q 4g*** 
(0.10) 

-0.07 

(0.18) 

College D 

Support 

Advising 

0.14 

(0.25) 

0.46*** 

(0.07) 

0.05 

(0.13) 

-0.07 

(0.17) 

0.32 

(0.29) 

College N 

Support 

Advising 

-1.08* 

(0.48) 

-0.04 

(0.13) 

0.29 

(0.24) 

-0.17 

(0.24) 

-0.65 

(0.34) 

College G 

Support 

Advising 

IN 

IN 

IN 

IN 

IN 

College 0 

Support 

Advising 

-0.38 

(0.39) 

0.20 

(0.14) 

0.60 

(0.37) 

-0.10 

(0.49) 

NA 

College 0 

Support 

Advising 

-0.83*** 

(0.23) 

0.01 

(0.08) 

-0.34 

(0.19) 

-0.21 

(0.22) 

NA 
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Appendix Table A.2 (continued) 


College 

Objective 

Approach 

Credits 
Earned 
Term 1 

GPA 
Term 1 

Persist 
to Term 2 

Pass English 
Gatekeeper 
by Term 2 

Pass Math 
Gatekeeper 
by Term 2 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

College B 

Support 

Study skills 

IN 

IN 

IN 

IN 

IN 

College C 

Support 11 

Study skills 

\ 40*** 

0 . 12 *** 

0.24** 

0.76*** 

0.81*** 




(0.13) 

(0.03) 

(0.09) 

(0.08) 

(0.09) 

College E 

Support 

Study skills 

0.12 

-0.04 

-0.23 

NA 

0.84** 




(0.39) 

( 0 . 11 ) 

(0.19) 


(0.32) 

College L 

Support 

Study skills 

0.01 

- 0 . 20 * 

- 0.22 

NA 

IN 




(0.32) 

(0.09) 

(0.17) 



College L 

Support 

Study skills 

-0.50*** 

0 07 *** 

-0.38*** 

0.09 

-1 93 *** 




(0.07) 

( 0 . 02 ) 

(0.05) 

(0.07) 

(0.26) 

College E 

Support 

T utoring 

1.08 

0.39 

IN 

IN 

IN 




(0.80) 

( 0 . 21 ) 




College H 

Support 

T utoring 

3 24*** 

0.07 

0.15 

NA 

0.10 




(0.71) 

(0.15) 

(0.30) 


(0.47) 

College H 

Support 

T utoring 

1.99* 

0.33 

0.62 

0.50 

NA 




( 0 . 86 ) 

(0.17) 

(0.35) 

(0.36) 


College C 

Support 11 

T utoring 

0.84*** 

0.00 

0.26*** 

0.58*** 

0.53*** 




(0.06) 

( 0 . 02 ) 

(0.04) 

(0.04) 

(0.05) 

College C 

High school 

NA 

IN 

IN 

IN 

IN 

IN 


support 
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Appendix Table A.2 (continued) 





Credits 
Earned 
Term 1 

GPA 
Term 1 

Persist 
to Term 2 

Pass English 
Gatekeeper 
by Term 2 

Pass Math 
Gatekeeper 
by Term 2 

College 

Objective 

Approach 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

College I 

High school 
support 

NA 

IN 

IN 

IN 

IN 

IN 

College F 

NA 

NA 

IN 

IN 

IN 

IN 

IN 

College F 

NA 

NA 

IN 

IN 

IN 

IN 

IN 

College H 

NA 

NA 

IN 

IN 

IN 

IN 

IN 

College J 

NA 

NA 

IN 

IN 

IN 

IN 

IN 

College M 

NA 

NA 

IN 

IN 

IN 

IN 

IN 


NOTES: NA = Not available. 

IN = insufficient sample size (N) for calculation. 

The table includes N = 46 focal strategies. Each cell shows the Equation (1) (for credits earned and GPA) or Equation (2) (for 
persistence and gatekeeper performance) (1, coefficient and standard error, where x l = 1 indicates treatment and x l = 0 indicates comparison 
group membership. Tables in Chapter 5 include only strategies with a valid coefficient for the given outcome under examination. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
a The comparison group was constructed from the data, based on the school’s definition of the target group. 



Comparison of Outcome Differences Between Participants and 
Nonparticipants in Individual Interventions 

The regression models summarized in Appendix Table A.2 were performed separately for each 
individual strategy. Each model examined students who were in the target group for the given 
intervention in their first semester and compared outcomes between those who were treated in 
their first semester and outcomes for those who were not. Regression models were performed 
only for the outcomes that were relevant to the given strategy. For example, for a math-oriented 
strategy, English enrollment and passing models would not be estimated; these models are 
replaced with the code “NA” (“Not Applicable”) in Appendix Table A.2. 

The multiple regression models control for term of entry (dummy-coded), gender, age 
(over/under 25), minority status, level of referral in each of the three developmental subjects, 
and whether the student was involved in another DEI intervention in the same term. The 
inclusion of term of entry as a control effectively compares treatment and comparison students 
within the same term, which is important, given that several strategies shifted the definition of 
the target population across terms. Other demographic variables available in the data set include 
whether the student was attending college for the first time or had previously attended another 
college, whether the student had been a dual-enrolled high school student, and whether the 
student was the first in the family to attend college. Unfortunately, some schools did not collect 
these data elements and were unable to include them in the DEI data set. Accordingly, their 
inclusion in the regression models would result in a more complex, more controlled model for 
some strategies and in a simpler model for other strategies. In order to remain consistent in the 
method across schools and strategies, covariates were limited to those that were reported by all 
schools. The DEI data set also includes more precise categorizations of ethnicity (that is, white, 
Hispanic, black, Asian, or other). These ethnic classifications were collapsed into a single 
minority-status dummy variable due to convergence issues with the logistic regression models, 
as discussed in more detail below. 

To ensure sufficient data to allow for reliable treatment estimates, regression models 
were perfonned only when at least 50 treatment and 50 comparison observations were available 
to the model. (Further restrictions were also placed on logistic regression models; see below.) 
For continuous outcomes (credits earned and continuous GPA), analyses were perfonned 
separately within each strategy, using a classic linear multiple regression model: 

v = Po +^iP i + Aj3 + s (1) 

Where x ] indicates the treatment dummy (with the value 1 indicating treatment and 0 indicating 
comparison group membership) and X represents the vector of covariates. The value of the p i 
coefficient and its standard error are shown in Appendix Table A.2. These coefficients can be 
interpreted in terms of the original scale of the outcome variable. For example, cumulative GPA 



varies from 0.0 to 4.0. A coefficient of 0.10 for first-term GPA would indicate that the treatment 
is associated with an increase of a tenth of a point in GPA. As an illustration, comparison 
students might have a first-tenn GPA of 2.0, while treatment students might have a first-term 
GPA of2.1. 

For binary outcomes (persistence, math gatekeeper enrollment and passing, and English 
gatekeeper enrollment and passing), analyses were performed separately within each strategy, 
using an analog of Equation (1) in the logistic regression context: 

logit(y) = Po +xiP i + AJ3 + s (2) 

The resulting Pi is a logit coefficient. As a rough rule of thumb, these coefficients can be 
interpreted by dividing the coefficient by 4; the resulting number gives a sense of the maximum 
increase (or decrease) in the probability of the outcome given a movement from the comparison 
to the treatment group. For example, a logit of 0.50 for student persistence into the second term, 
when divided by 4, yields 0.125, or 12.5 percent. As an illustration, if comparison students have 
a 50 percent chance of persisting to Term 2, then treatment students would have approximately 
a 62.5 percent chance of doing so. 3 

Logistic regressions can be problematic when sample sizes are low; if the data matrix is 
too sparse (for example, it has few or zero observations for certain cells), then the model cannot 
properly converge, and the resulting estimates cannot be trusted. In initial test runs with the 
logistic regression models, a large proportion of models failed to converge. Simplifying the set 
of ethnicity variables to a single minority-status dummy code ameliorated the sparsity problem 
for several smaller-sample strategies. The remaining nonconvergent models’ sparsity problems 
seemed centered on few or no observations in one or more of the treatment-by-outcome cells. 
Instituting a rule of a minimum of 5 observations in each treatment-by-outcome cell eliminated 
the remaining strategies from consideration in terms of logistic regression modeling. In Appen- 
dix Table A.2, these models are noted as “IN,” as having Insufficient N. 

For both types of models, sample sizes vary widely across strategies: Some strategies 
served thousands of students across the time frame under study, while others served just a few. 
In general, sample sizes for the regressions that are summarized in Appendix Table A.2 are 
smaller than the totals for the strategies, as the regressions include only first-semester target 
students while the strategy counts include both first-semester and continuing target students. 
Typically, however, the bulk of target students were first-semester students. 


3 This rule of thumb works best when the dependent variable hovers near a 50 percent chance for the posi- 
tive outcome. If the chance of the positive outcome is very low (for example, 5 percent) or very high (for 
example, 95 percent), the change in probability associated with a given logit will be much less than that given 
by the rule of thumb. 
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In addition to variation in sample size across strategies, sample sizes for a given strategy 
also vary across the time frame of the outcome examined. That is, all first-semester treatment 
and comparison students are included in the Tenn 1 outcomes, while only students with next- 
term follow-up data can be examined for the Tenn 2 outcomes (thus excluding the fall 2011 
cohort, for whom winter/spring 2012 data are not yet available). 

In general, the sample sizes for each regression model can be intuited from the size of 
the coefficient’s standard enor, relative to the scale of the outcome under study. When consider- 
ing the same outcome, those strategies with the smallest standard enors tend to have large 
sample sizes (often in the thousands), while those strategies with the largest standard enors tend 
to have small sample sizes (often nearing the minimum sample size of 100). This, of course, 
illustrates the inverse relationship between sample size and the power to detect an effect: Even a 
large estimated effect may not be statistically significant if the sample size is small and, thus, the 
standard enor is large. 

Note that no adjustments are applied for multiple significance testing across the regres- 
sion models. Pragmatically, given the number of models perfomied, it is unlikely that any 
coefficient would be significant if such adjustments were applied. Results that are significant at 
an unadjusted p-value of less than or equal to 5 percent are indicated in Appendix Table A.2 in 
order to provide an overall portrait of the areas where positive or negative results seem to be 
clustered, rather than to attempt to judge the significance of any particular model or program. 

Appendix Tables A.3 and A.4 present summary information about the relationship be- 
tween the direction of outcome differences (whether they are positive, negative, or neutral) and 
the extent of participation in the focal strategies. Appendix Table A.3 measures extent of 
participation in the focal strategies against the goals that colleges set for themselves. Appendix 
Table A.4 instead measures participation in the focal strategies as a percentage of the target 
populations for these strategies. 


Comparison of College-Level Outcome Differences Before and 
After Implementation of the DEI 

Multiple regression was also used to estimate the change in outcomes subsequent to the imple- 
mentation of the DEI, separately for each college. Using Equation (1) for continuous outcomes 
and Equation (2) for binary outcomes, x\ indicates the timing of the student’s entry into college, 
where the value 1 indicates entry before DEI implementation and the value 0 indicates entry 
after DEI implementation. It is difficult to determine with clear certainty when “DEI implemen- 
tation” was achieved. Before fall 2009, colleges had been implementing AtD-related develop- 
mental education refonns and programming, but they often did so on a relatively small scale. 
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The Developmental Education Initiative 
Appendix Table A.3 

Outcome Differences Associated with Participation in the 
DEI Focal Strategies, by Proportion of 
2010-2011 Student Participation Goals Attained 


Outcome 

Measured 

and 

Proportion 

Percentage of Focal Strategies for Which 
Outcome Difference Is: 


Number of 
Strategies for 
Which 
Outcome Was 
Measured 

Attained Positive 

Neutral 

Negative 

Total 

Credits earned in Term 1 





26 

Less than 25% 

44.4 

44.4 

11.1 

100.0 


25 to 50% 

28.6 

71.4 

0.0 

100.0 


More than 50% 

60.0 

30.0 

10.0 

100.0 


GPA earned in Term 1 





26 

Less than 25% 

33.3 

55.6 

11.1 

100.0 


25 to 50% 

14.3 

85.7 

0.0 

100.0 


More than 50% 

40.0 

60.0 

0.0 

100.0 


Persistence to next term 





23 

Less than 25% 

14.3 

71.4 

14.3 

100.0 


25 to 50% 

16.7 

83.3 

0.0 

100.0 


More than 50% 

50.0 

50.0 

0.0 

100.0 


Pass English bv end of Term 2 





13 

Less than 25% 

0.0 

100.0 

0.0 

100.0 


25 to 50% 

50.0 

50.0 

0.0 

100.0 


More than 50% 

71.4 

28.6 

0.0 

100.0 


Pass math bv end of Term 2 





17 

Less than 25% 

20.0 

60.0 

20.0 

100.0 


25 to 50% 

25.0 

75.0 

0.0 

100.0 


More than 50% 

37.5 

62.5 

0.0 

100.0 



NOTE: The table shows outcomes associated with participation in the focal strategies; it cannot be 
inferred that participation caused these outcomes. 
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The Developmental Education Initiative 
Appendix Table A.4 

Outcome Differences Associated with Participation in the 
DEI Focal Strategies, by Proportion of Target Population 
Served in Fall 2011 


Outcome 

Measured 

and 

Proportion 

Percentage of Focal Strategies for Which 
Outcome Difference Is: 

Number of 
Strategies for 
Which 
Outcome Was 
Measured 

Attained 

Positive 

Neutral 

Negative 

Total 

Credits earned in Term 1 





30 

Less than 25% 

56.3 

37.5 

6.3 

100.0 


25 to 50% 

22.2 

55.6 

22.2 

100.0 


More than 50% 

60.0 

40.0 

0.0 

100.0 


GPA earned in Term 1 





30 

Less than 25% 

31.3 

62.5 

6.3 

100.0 


25 to 50% 

11.1 

88.9 

0.0 

100.0 


More than 50% 

60.0 

40.0 

0.0 

100.0 


Persistence to next term 





27 

Less than 25% 

25.0 

75.0 

0.0 

100.0 


25 to 50% 

33.3 

50.0 

16.7 

100.0 


More than 50% 

40.0 

60.0 

0.0 

100.0 


Pass English bv end of Term 2 





18 

Less than 25% 

60.0 

40.0 

0.0 

100.0 


25 to 50% 

25.0 

75.0 

0.0 

100.0 


More than 50% 

50.0 

50.0 

0.0 

100.0 


Pass math bv end of Term 2 





17 

Less than 25% 

22.2 

77.8 

0.0 

100.0 


25 to 50% 

20.0 

60.0 

20.0 

100.0 


More than 50% 

66.7 

33.3 

0.0 

100.0 



NOTE: The table shows outcomes associated with participation in the focal strategies; it cannot be 
inferred that participation caused these outcomes. 
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During the DEI “kickoff semester” of fall 2009, most colleges were engaged in planning in 
tenns of program scale-up, policy changes, or the implementation of entirely new programs, 
with more substantial implementation of these plans occurring in spring or fall 2010. Accord- 
ingly, cohorts entering in fall 2009 or earlier were defined as “pre-DEI” cohorts, and those 
entering in spring 2010 or later were defined as “post-DEI” cohorts. 

Although the DEI focused primarily on developmental students, this analysis includes 
all new students, for three reasons. First, avoidance strategies are designed to move some 
students from developmental to college-ready. By including all new students, this analysis can 
account for the outcomes of those who were moved into the college-ready category; in contrast, 
if only developmental students were included, the avoidance strategies would likely remove the 
best-prepared or highest-level developmental students from the sample, potentially dampening 
outcomes. Second, many DEI strategies and policies affected incoming college-ready students 
as well as developmental students. And third, for most schools, the majority of the entering 
population is classified as developmental; thus, the inclusion of college-ready students — even 
if DEI had no effect on these students — should not “water down” the results much. 

For these multiple regression equations, the vector of covariates X includes entry in a 
fall term versus spring term, gender, age, minority status, and level of developmental referral for 
each of the three developmental subjects. The Pi coefficient for each college and its associated 
standard error are shown in Appendix Table A.5. As with the individual intervention models, 
the college-level models vary in their sample sizes, as some colleges had large enrollments and 
others had smaller enrollments. However, as Table 1.1 (Chapter 1) shows, all the colleges had 
enrollments of several thousands of students; thus, the variations in sample size have less 
substantial impacts on standard errors and on the power of the model to detect an effect. 

Given that there are multiple colleges with sufficient data to run the overall regression 
model, an analyst might also consider estimating a multilevel model (also known as a random- 
effects model or a hierarchical linear model ) across the entire set of colleges and students, 
releasing the pre- versus post-DEI coefficient to vary across colleges, and using the college- 
specific variation to estimate the individual college’s coefficient. With a large number of 
colleges, the multilevel approach would be both more efficient and more reliable. However, 
given a small number of colleges (only 14 reported developmental referral data), there are 
insufficient college-level units to satisfy the assumption of normality in the distribution of 
college-level coefficients. Accordingly, in this report, each college’s coefficient was estimated 
in a separate model. 
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The Developmental Education Initiative 
Appendix Table A.5 

Differences in Outcomes Between Students Entering Each College 
Before and After DEI Implementation 


College 

Credits 
Earned 
Term 1 

GPA 
Term 1 

Persist 
to Term 2 

Pass English 
Gatekeeper 
by Term 2 

Pass Math 
Gatekeeper 
by Term 2 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

Coefficient 

(SE) 

College A 

0.67*** 

0.06* 

-0.28*** 

-0 27*** 

-0.17* 


(0.08) 

(0.03) 

(0.05) 

(0.06) 

(0.08) 

College B 

0.50*** 

0.08** 

0.02 

0.04 

-0.04 


(0.08) 

(0.03) 

(0.05) 

(0.07) 

(0.09) 

College C 

1 24*** 

-0 09*** 

0.20*** 

0.18*** 

-0.06* 


(0.03) 

(0.01) 

(0.02) 

(0.02) 

-0.03 

College D 

0.26*** 

0.16*** 

0.16*** 

0.10** 

q 27*** 


(0.07) 

(0.02) 

(0.04) 

(0.04) 

(0.06) 

College E 

-0 92*** 

0.03 

0.00 

Q 14*** 

0.17** 


(0.06) 

(0.02) 

(0.03) 

(0.04) 

(0.06) 

College F 

-0.04 

-0.17** 

0.38** 

-0.16 

-1.07* 


(0.23) 

(0.06) 

(0.12) 

(0.14) 

-0.44 

College G 

-0.06 

-0.05 

0.32** 

0.06 

-0.08 


(0.19) 

(0.04) 

(0.10) 

(0.08) 

(0.09) 

College H 

0.39** 

-0.12*** 

0.03 

0.03 

0.02 


(0.15) 

(0.03) 

(0.07) 

(0.07) 

(0.09) 

College I 

0.05 

-0.12*** 

-0.02 

0.22*** 

Q 14*** 


(0.06) 

(0.01) 

(0.03) 

(0.03) 

(0.04) 

College J 

0.12 

-0.04 

0.02 

0.15* 

-0.04 


(0.13) 

(0.03) 

(0.07) 

(0.07) 

(0.12) 

College K 

-0.03 

-0. 14*** 

0.08** 

0.25*** 

-0.20*** 


(0.04) 

(0.01) 

(0.03) 

(0.03) 

(0.04) 

College L 

-0 40*** 

-0.20*** 

0.10*** 

-0.10*** 

-0.28*** 


(0.03) 

(0.01) 

(0.02) 

(0.02) 

(0.03) 

College M 

-0.74*** 

-0.25*** 

-2 17*** 

-0.19* 

0.12 


(0.17) 

(0.04) 

(0.09) 

(0.09) 

(0.21) 

College N 

0.36* 

0.14** 

-0.14 

0.15 

-0.09 


(0.15) 

(0.04) 

(0.08) 

(0.09) 

(0.12) 


(continued) 
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Appendix Table A.5 (continued) 



Credits 



Pass English 

Pass Math 


Earned 

GPA 

Persist 

Gatekeeper 

Gatekeeper 


Term 1 

Term 1 

to Term 2 

by Term 2 

by Term 2 


Coefficient 

Coefficient 

Coefficient 

Coefficient 

Coefficient 

College 

(SE) 

(SE) 

(SE) 

(SE) 

(SE) 

College 0 

0.96*** 

(0.05) 

0.02 

(0.02) 

0.03 

(0.04) 

0 32 *** 
(0.04) 

0.54*** 

(0.06) 


SOURCE: MDRC and CCRC calculations from colleges' reports to JBL. 

NOTES: Each cell shows the Equation (1) (for credits earned and GPA) or Equation (2) (for 
persistence and gatekeeper performance) ft, coefficient and standard error, where x l = 1 indicates 
post-DEI entry and x 1 = 0 indicates pre-DEI entry. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 




